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(57) On the transmission side, a background picture 
and objects #1 to #3 are transmitted at a transmission 
rate R/4 each. On the reception side, a picture com- 
posed of the objects #1 to #3 and the background pictu re 
is displayed with a particular spatial resolution and a par- 
ticular temporal resolution. In this case, on the reception 
' side, when the object #1 is dragged at particular time t1 , 



on the transmission side, as. shown in Fig. 16 (A), the 
transmission of the background picture and the objects 
#2 and #3 is stopped. Only the object #1 is transmitted 
at the transmission rate R of the transmission path. 
Thus, a picture of which the spatial resolution of the ob- 
ject #1 dragged is improved is displayed at the sacrifice 
of the temporal resolution of the picture. 
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Description 

Technical Field 

[0001] The present invention relates to a transmitting 
apparatus, a transmitting method, a receiving appara- 
tus, a receiving method, a transmitting and receiving ap- 
paratus, a transmitting and receiving method, a record 
medium, and a signal. In particular, the present inven- 
tion relates to a transmitting apparatus, a transmitting 
method, a receiving apparatus, a receiving method, a 
transmitting and receiving apparatus, a transmitting and 
receiving method, a record medium, and a signal that 
allow for example picture data to be transmitted at a lim- 
ited transmission rate (with a limited transmission band) 
and a picture with a high spatial resolution to be dis- 
played. 

Related Art 

[0002] For example, a related art reference of Japa- 
nese Patent Laid Open Publication No. HEI 10-112856 
discloses a picture transmitting apparatus. In the picture 
transmitting apparatus, a transmission side transmits 
picture data of a particular area of a picture and picture 
data of the other area of the picture are transmitted with 
different information amounts corresponding to a com- 
mand issued by a reception side. The reception side dis- 
plays the picture in the area that contains a designated 
point with a higher spatial resolution (the resolution in 
the spatial direction) and the picture in the other area 
with a lower spatial resolution. 

[0003] In other words, when the transmission side 
transmits picture data to the reception side through a 
transmission path, the transmission side cannot trans- 
mit picture data at a higher data rate than that of the 
transmission path. Thus, when the reception side dis- 
plays a picture on a real time basis, the transmission 
side should transmit picture data to the reception side 
at a transmission rate corresponding to the transmission 
path. As a result, if the transmission rate is not sufficient, 
the spatial resolution in the spatial direction of a picture 
displayed on the reception side totally deteriorates. 
[0004] To solve such a problem, in the picture trans- 
mitting apparatus of Japanese Patent Laid Open Publi- 
cation No. HEI 10-112856, picture data in one area of a 
picture and picture data in other area of the picture are 
transmitted with different information amounts. The pic- 
ture in the area that contains a designated point and the 
picture in the other area are displayed with a high spatial 
resolution and a low spatial resolution, respectively, on 
the reception side. 

[0005] In other words, in the picture transmitting ap- 
paratus of Japanese Patent Laid Open Publication No. 
HEI 10-112856, the spatial resolution of a portion that 
the user wants to see in detail is improved at the sacrifice 
of the spatial resolution of the other portion. 
[0006] However, in the picture transmitting apparatus 



of Japanese Patent Laid Open Publication No. HEI 
10-112856, since the spatial resolution of the portion 
that the user wants to see in detail is improved at the 
sacrifice of the spatial resolution of the other portion, the 
5 spatial resolution of the portion that the user wants to 
see in detail is improved for only the sacrificed special 
resolution. 

[0007] In addition, when the transmission rate of the 
transmission path is very low, if the spatial resolution of 

10 the portion that the user wants to see in detail is im- 
proved at the sacrifice of the spatial resolution of the 
other portion, the spatial resolution of the other portion 
terribly deteriorates. In the worst case, the user cannot 
clearly see the other portion. 

15 [0008] The present invention is made from the above- 
described point of view. An object of the present inven- 
tion is to allow the spatial resolution of a picture to be 
more improved corresponding to a preference of the us- 
er. 

20 

Disclosure of the Invention 

[0009] A first transmitting apparatus of the present in- 
vention is a transmitting apparatus for transmitting data 

25 to a receiving apparatus, comprising a receiving means 
for receiving control information transmitted from the re- 
ceiving apparatus, a controlling means for controlling 
the resolutions in at least two directions of the temporal 
direction, the spatial direction, and the level direction of 

30 the data transmitted to the receiving apparatus corre- 
sponding to the control information, and a transmitting 
means for transmitting the data of which the resolutions 
in at least two directions have been controlled corre- 
sponding to the control information to the receiving ap- 

35 paratus. 

[0010] A receiving apparatus of the present invention 
is a receiving apparatus for receiving data transmitted 
from a transmitting apparatus, comprising a transmitting 
means for transmitting control information to the trans- 

40 mitting apparatus that controls resolutions in at leasttwo 
directions of the temporal direction, the spatial direction, 
and the level direction of the data corresponding to the 
control information, a receiving means for receiving the 
data transmitted from the transmitting apparatus, the 

45 resolutions of the data having been controlled in at least 
two directions of the data, and an outputting means for 
outputting the data received by the receiving means. 
[0011] A transmitting and receiving apparatus of the 
present invention is a transmitting and receiving appa- 

50 ratus having a transmitting apparatus for transmitting 
data and a receiving apparatus for receiving the data, 
wherein the transmitting apparatus comprises a control 
information receiving means for receiving control infor- 
mation transmitted from the receiving apparatus, acon- 

55 trolling means for controlling the resolutions in at least 
two directions of the temporal direction, the spatial di- 
rection, and the level direction of the data transmitted to 
the receiving apparatus corresponding to the control in- 
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Brief Description of Drawings 

[0029] Fig. 1 is a schematic diagram showing an ex- 
ample, of the structure of a transmitting system accord- 
ing to an embodiment of the present invention. 
[0030] Fig. 2 is a detailed schematic diagram showing 
an example of a first structure of the transmitting system 
shown in Fig. 1 . 

[0031] Fig. 3 is a block diagram showing an example 
of the structure of a transmitting apparatus (terminal 
unit) 1 shown in Fig. 1 . 

[0032] Fig. 4 is a flow chart for explaining the process 
of the transmitting apparatus shown in Fig. 3. 
[0033] Fig. 5 is a block diagram showing an example 
of the structure of a receiving apparatus (terminal unit) 
2 shown in Fig. 1. 

[0034] Fig. 6 is a flow chart for explaining the process 
of the receiving apparatus 2 shown In Fig. 5. 
)[0035] Fig. 7 is a block diagram showing an example 
} of the structure of a transmitting process portion 16 
shown in Fig. 3. 

[0036] Fig. 8 is a block diagram showing an example 
of the structure of an encoding portion 31 shown in Fig. 
7. 

[0037] Fig. 9 is a schematic diagram for explaining hi- 
erarchical encoding / decoding. 

[0038] Fig. 10 is a flow chart for explaining a transmit- 
ting process of a transmitting process portion 1 6 shown 
in Fig. 7. 

[0039] Fig. 1 1 is a block diagram showing an example 
of the structure of a receiving process portion 21 shown 
in Fig. 5. 

[0040] Fig. 12 is a block diagram showing an example 
of the structure of a decoding portion 53 shown in Fig. 
11. 

[0041] Fig. 13 is a block diagram showing an example 
of the structure of a combining process portion 22 shown 
in Fig. 5. 

^[0042] Fig. 14 is a flow chart for explaining a combin- 
^ ing process of the combining process portion 22 shown 
in Fig. 13. 

[0043] Fig. 1 5 is a schematic diagram showing an ex- 
ample of a display of a picture of a picture outputting 
portion 23 shown in Fig. 5. 

[0044] Fig. 16 is a schematic diagram for explaining 
the relation between the spatial resolution and the tem- 
poral resolution of a picture transmitted from the trans- 
mitting apparatus 1 to the receiving apparatus 2 shown 
in Fig. 1 . 

[0045] Fig. 1 7 is a block diagram showing an example 
of the structure of an object extracting portion 1 4 shown 
in Fig. 3. 

[0046] Fig. 18 is a schematic diagram for explaining 
the process of an initial area dividing portion 83 shown 
in Fig. 17. 

[0047] Fig. 19 is a schematic diagram for explaining 
the process of an area merging portion 84 shown in Fig. 
17. . 



[0048] Fig. 20 is a schematic diagram for explaining 
the processes of a merged area process portion 85 and 
a separated area process portion 86 shown in Fig. 17. 
[0049] Fig. 21 is a flow chart for explaining the object 
5 extracting process of the object extracting portion 14 
shown in Fig. 17. 

[0050] Fig. 22 is a flow chart for explaining the detail 
" of the area merging process at step S43 shown in Fig. 
21. 

10 [0051 ] Fig. 23 is a flow chart for explaining the detail 
of the merged area process at step S44 shown in Fig. 
21. 

[0052] Fig. 24 is a flow chart for explaining the detail 
of the separated area process at step S44 shown in Fig. 
^ 21. 

[0053] Fig. 25 is a block diagram showing an example 
of the structure of a controlling portion 35 shown in Fig. 
7. 

[0054] Fig. 26 is a schematic diagram for explaining 
20 a feature amount of an object. 

[0055] Fig. 27 is a flow chart for explaining the detail 
of the process of the controlling portion 35 shown in Fig. 
25. 

[0056] Fig. 28 is a detailed schematic diagram show- 
25 ing an example of a second structure of the transmitting 
system shown in Fig. 1 . 

[0057] Fig. 29 is a block diagram.showing an example 
of the structure of a transmitting apparatus 1 shown in 
Fig. 28. 

30 [0058] Fig. 30 is a flow chart for explaining the process 
of the transmitting apparatus 1 shown in Fig. 29. 
[0059] Fig. 31 is a block diagram showing an example 
of the structure of a receiving apparatus 2 shown in Fig. 
28. 

35 [0060] Fig.32isaflowchartforexplainingtheprocess 
of the receiving apparatus 2 shown in Fig. 31 . 
[0061 ] Fig. 33 is a block diagram showing an example 
of the structure of a transmitting process portion 1016 
shown in Fig. 29. 

40 [0062] Fig. 34 is a flow chart for explaining the trans- 
mitting process of the transmitting process portion 1016 
shown in Fig. 33. 

[0063] Fig. 35 is a block diagram showing an example 
of the structure of a combining process portion shown 
45 in Fig. 31. 

[0064] Fig. 36 is a flow chart for explaining the com- 
bining process of a combining process portion shown in 
Fig 35. 

[0065] Fig. 37 is a schematic diagram showing a real 
50 structure of an object extracting portion 1014 shown in 
Fig. 29. 

[0066] Fig. 38 is a flow chart for explaining the extract- 
ing process for a moving object picture or a still object 
picture. 

55 [0067] Fig. 39 is a flow chart for explaining a still area 
and moving area determining process. 
[0068] Fig. 40 is a schematic diagram for explaining 
a calculating method for the difference between frames. 
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[0069] Fig. 41 is a flow chart for explaining a continu- 
ous click determining process. 

[0070] Fig. 42 is a schematic diagram for explaining 
an object number assigning method. 
[0071] Fig. 43 is a flow chart for explaining a still object 
combining process. 

[0072] Fig. 44 is a flow chart for explaining a moving 
object combining process. . 

[0073] Fig. 45 is a flow chart for explaining an object 
extracting process. 

[0074] Fig. 46 is a schematic diagram for explaining 
an object extracting method. 

[0075] Fig. 47 is a block diagram showing an example 
of another structure of the combining process portion 
1022 shown in Fig. 31. 

[0076] Fig. 48 is a block dia gram showing an example 
of another structure of the transmitting apparatus 1 
shown in Fig. 28. 

[0077] Fig. 49 is a block diagram showing an example 
of the structure of a change determining and categoriz- 
ing portion 240 shown in Fig. 48. 

[0078] Fig. 50 is a flow chart for explaining the inter- 
ested object area changing process and the interested 
object area categorizing process of the change deter- 
mining and categorizing portion 240. 
[0079] Rg. 51 is a flow chart for explaining the interest 
change determining process. 

[0080] Fig. 52 is a schematic diagram showing a de- 
tailed example of a third structure of the transmitting sys- 
tem shown in Fig. 1 . 

[0081] Fig. 53 is a block diagram showing an example 
of the structure of a charging server 4 shown in Fig. 52. 
[0082] Fig. 54 is a flow chartforexplaining the process 
of the charging server 4 shown in Fig. 53. 
[0083] Fig. 55 is a block diagram showing an example 
of the structure of a computer according to an embodi- 
ment of the present invention. 

Best Modes for Carrying out the Invention 

[0084] Fig. 1 shows the structure of a picture trans- 
mitting system (the system is a substance as a logical 
set of a plurality of devices regardless of whether or not 
each device is disposed in one housing) according to 
an embodiment of the present invention. 
[0085] The transmitting system is composed of at 
least two terminal units 1 and 2. In the relation between 
the terminal units 1 and 2, one is a transmitting appara- 
tus and the other is a receiving apparatus. A picture (pic- 
ture data) is transmitted from the transmitting apparatus 
to the receiving apparatus through a transmission path 
3. 

[0086] According to the embodiment of the present in- 
vention, for.example, it is assumed that the terminal unit 
1 is a transmitting apparatus and the terminal unit 2 is 
a receiving apparatus and that picture data is transmit- 
ted and received therebetween. Hereinafter, the termi- 
nal unit 1 or 2 may be referred to as transmitting appa- 



ratus 1 or receiving apparatus 2, respectively. 
[0087] In this case, the transmitting apparatus 1 trans- 
mits picture data to the receiving apparatus 2 through 
the transmission path 3. The receiving apparatus 2 re- 
5 ceives picture data from the transmitting apparatus 1 . 
The picture data is displayed on a picture outputting por- 
tion 23 (see Fig. 5) that will be described later. The pic- 
ture outputting portion 23 is composed of for example a 
liquid crystal display or a CRT (Cathode Ray Tube). In 
addition, the receiving apparatus 2 transmits control in- 
formation for controlling the spatial resolution in the spa- 
tial direction and the temporal resolution in the temporal 
direction of the picture displayed by the picture output- 
ting portion 23 to the transmitting apparatus 1 through 
the transmission path 3. 

[0088] The transmitting apparatus 1 receives the con- 
trol information from the receiving apparatus 2 and con- 
trols the transmission of the picture data corresponding 
to the control information so that the spatial resolution 
and the temporal resolution of the picture displayed by 
the receiving apparatus 2 are changed while a prede- 
termined condition is satisfied. 

[0089] As the transmitting apparatus 1 andthe receiv- 
ing apparatus 2, terminal units such as PHS (Personal 
Handy-phone System) (trademark) units and portable 
terminal units such as portable telephone units can be 
used. When the PHS units are used, the transmission 
path 3 has a frequency band of 1 895.1 500 to 1 905.9500 
MHz and the transmission rate thereof is 128 kbps (Bit 
Per Second). 

[0090] Fig. 2 shows an example of a first structure of 
the picture transmitting system shown in Fig. 1 in the 
case that portable terminal units such as PHS units and 
portable telephone units are used as the transmitting ap- 
paratus 1 andthe receiving apparatus 2 shown in Pig. 1 . 
[0091] According to the embodiment shown in Fig. 2, 
the transmission path 3 shown in Fig, 1 is composed of 
a radio base station 3-1 or 3-2 and an exchange station 
3-3 that is for example a telephone station that connects 
the radio base stations 3-1 and 3-2. The radio base sta- 
tion 3-1 or 3-2 transmits and receives a radio signal to / 
from the terminal unit 1 or 2. Between the terminal units 
1 and 2, each of them can transmit a signal to the other 
through the transmission path 3 composed of the radio 
base stations 3-1 and 3-2, the charging server 4, and so 
forth and receive a signal from the other. The type of the 
base station 3-1 may be the same as or different from 
the type of the base station 3-2. 

[0092] Referring to Fig. 2, the terminal unit 1 compris- 
es a video camera portion 1 -1 , a displaying portion 1 -2, 
a key portion 1-3, a speaker 1-4, and a microphone 1-5. 
The video camera portion 1-1 has an image pickup de- 
vice and an optical system that can photograph a mov- 
ing picture. The displaying portion 1-2 can display char- 
acters, symbols, pictures, and so forth. The key portion 
1-3 is operated when a telephone number, a character, 
a command, or the like is input. The speaker 1 -4 outputs 
a sound. The microphone 1 -5 inputs a sound. Likewise, 



75 



20 



25 



30 



35 



40 



45 



50 



11 EP1 120 968 A1 12 



the terminal unit 2 comprises a video camera portion 
2-1 , a displaying portion 2-2, a key portion 2-3, a speak- 
er 2-4, and a microphone 2-5 that have the same struc- 
tures as the video camera portion 1-1, the displaying 
portion 1-2, the key portion 1-3, the speaker 1-4, and 
the microphone 1-5, respectively. 
[0093] Between the terminal units 1 and 2, audio sig- 
nals collected by the microphones 1 -5 and 1 -6 are trans- 
mitted and received. In addition, picture data photo- 
graphed by the video camera portions 1-1 and 2-1 can 
be transmitted and received. Thus, the displaying por- 
tion 1-2 of the terminal unit 1 can display picture data 
obtained by the video camera portion 2-1 of the terminal 
unit 2. Likewise, the displaying portion 2-2 of the termi- 
nal unit 2 can display picture data obtained by the video 
camera portion 1 -1 of the terminal unit 1 . 
[0094] In other words, along with necessary informa- 
tion such as a frame rate, picture data photographed by 
Uhe video camera portion 1-1 of the transmitting appa- 

\ ratus 1 is transmitted to the receiving apparatus 2 
through the transmission path 3 composed of the base 
stations 3-1 and 3-2 and the exchange station 3-3. The 
receiving apparatus 2 receives picture data from the 
transmitting apparatus 1 and displays (a picture corre- 
sponding to) the received picture data on the displaying 
portion 2-1 composed of for example a liquid crystal dis- 
play. On the other hand, the receiving apparatus 2 trans- 
mits control information for controlling the spatial reso- 
lution and the temporal resolution of a picture displayed 
on the displaying portion 2-1 to the transmitting appara- 
tus 1 through the transmission path 3. The transmitting 
apparatus 1 receives control information from the re- 
ceiving apparatus 2 and controls the transmission of the 
picture data corresponding to the control information so 
that the spatial resolution and the temporal resolution of 
the picture displayed by the receiving apparatus 2 are 
changed while a predetermined condition is satisfied. 
[0095] Next, Fig. 3 shows an example of the structure 
- of the transmitting apparatus 1 shown in Fig. 2. 

) [0096] A picture inputting portion 11 corresponds to 
the video camera portion 1-1 shown in Fig. 2. The pic- 
ture inputting portion 11 photographs a particular object 
and outputs the picture thereof to a pre-process portion 
12. The pre-process portion 12 is composed of a back- 
ground picture extracting portion 13, an object extract- 
ing portion 14, and an additional Information calculating 
portion 15. The pre-process portion 12 performs the pre- 
process for the picture that is input from the picture in- 
putting portion 11 and supplies the resultant picture to 
a transmitting process portion 16. 
[0097] In other words, the background picture extract- 
ing portion 13 extracts a so-called background picture 
from the picture supplied from the picture inputting por- 
tion 11 and supplies the extracted background picture 
to the transmitting process portion 16. In addition, the 
background picture extracted by the background picture 
extracting portion 13 is also supplied to the object ex- 
tracting portion 14 and the additional information calcu- 



lating portion 15. 

[0098] As a method for extracting the background pic- 
ture, the appearance frequencies of pixel values of pix- 
els at the same spatial position of a plurality of succes- 

5 sive frames (for example, the current frame and the past 
1 0 frames) are obtained. A pixel value with the highest 
frequency may be treated as a background picture at 
the position. Alternatively, the average value (moving 
average value) of pixel values at the same position of a 

10 plurality of frames may be obtained as a background pic- 
ture at the position. 

[0099] The object extracting portion 14 performs for 
example a subtraction of the background picture ex- 
tracted by the background picture extracting portion 13 

15 from the picture supplied from the picture inputting por- . 
tion 11 , obtains a so-called foreground picture, and sup- 
plies the foreground picture to the transmitting process 
portion 1 6. When a picture that is input from the picture 
inputting portion 11 contains a plurality of substances as 

20 foreground pictures, the object extracting portion 1 4 ex- 
tracts theforeground picture corresponding to each sub- 
stance and supplies the extracted foreground picture to 
the transmitting process portion 1 6. In addition, the fore- 
ground picture extracted by the object extracting portion 

25 14 is also supplied to the additional information calcu- 
lating portion 15. Hereinafter, a foreground picture cor- 
responding to each substance may be referred to as ob- 
ject. 

[0100] The additional information calculating portion 

30 15 detects a background picture moving vector and an 
object moving vector. The background picture moving 
vector represents the motion of a background picture 
extracted by the background picture extracting portion 
13 (the motion of a background picture corresponds to 

35 the motion of the photographing direction of the picture 
inputting portion 11). The object moving vector repre- 
sents the motion of an object extracted by the object ex- 
tracting portion 14. In addition, the additional information 
calculating portion 15 supplies position information and 

40 so forth of an object in a frame as additional information 
supplied from the object extracting portion 14 to the 
transmitting process portion 16. In other words, when 
the object extracting portion 14 extracts an object, the 
object extracting portion 14 also extracts information 

^5 about the object such as position information and so 
forth of the object and supplies the information to the 
additional information calculating portion 15. The addi- 
tional information calculating portion 15 also outputs the 
position information and so forth as additional informa- 

so tion. 

[01 01 ] The transmitting process portion 16 multiplex- 
es the background picture extracted by the background 
picture extracting portion 1 3, the object extracted by the 
object extracting portion 14, and the additional informa- 
55 tion obtained by the additional information calculating 
portion 15 and transmits the resultant multiplexed data 
to the receiving apparatus 2 through the transmission 
path 3 at a data rate thereof. The transmitting process 
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portion 16 receives control information transmitted from 
the receiving apparatus 2 through the transmission path 
3 and controls the transmission of. the background pic- 
ture, the object, and the additional information corre- 
sponding to the control information so that the spatial 
resolution and the temporal resolution of the picture dis- 
played by the receiving apparatus 2 are changed while 
a predetermined conditions is satisfied. 
[0102] Next, with reference to a flow chart shown in 
Fig. 4, the process of the transmitting apparatus 1 
shown in Fig. 3 will be described in brief. 
[0103] A picture that is output from the picture input- 
ting portion 1 1 is supplied to the pre-process portion 1 2. 
At step S1 , the pre-process portion 12 performs the pre- 
process for the picture. In other words, at step S1 , the 
background picture extracting portion 13 or the fore- 
ground picture extracting portion 14 extracts a back- 
ground picture or an object from the picture that is input 
from the picture inputting portion 11 , respectively, and 
supplies the extracted background picture or the ex- 
tracted object to the transmitting process portion 16. In 
addition, at step S1 , the additional information calculat- 
ing portion 1 5 obtains the above-described additional in- 
formation from the picture that is input from the picture 
inputting portion 11 and supplies the obtained additional 
information to the transmitting process portion 16. The 
transmitting process portion 16 transmits the back- 
ground picture, the foreground picture, and the addition- 
al information through the transmission path 3. There- 
after, the flow returns to step S1 . Thereafter, the trans- 
mitting apparatus 1 repeats the similar process. 
[01 04] Fig. 5 shows an example of the structure of the 
receiving apparatus 2 shown In Fig. 2. 
[01 05] Multiplexed data is transmitted from the trans- 
mitting apparatus 1 through the transmission path 3. 
The multiplexed data is received by the receiving proc- 
ess portion 21. The receiving process portion 21 sepa- 
rates the multiplexed data into a background picture, an 
object, and additional information and supplies them to 
the combining process portion 22. 
[0106] The combining process portion 22 combines 
the background picture, the object, and the additional 
information received from the receiving process portion 
21 into the original picture and supplies the combined 
picture to the picture outputting portion 23. The combin- 
ing process portion 22 changes the spatial resolution 
and the temporal resolution of the picture that is com- 
bined corresponding to high resolution information and 
so forth that will be described later. 
[01 07] The picture outputting portion 23 is composed 
of for example a liquid crystal display or a CRT. The pic- 
ture outputting portion 23 displays a picture that is output 
from the combining process portion 22. 
[0108] A control information inputting portion 24 out- 
puts control information to the combining process por- 
tion 22 and a control information transmitting portion 25. 
in other words, the control information inputting portion 
24 composes the key portion 2-3. The control informa- 



tion inputting portion 24 is composed of for example a 
pointing device such as a track ball. When the user des- 
ignates a particular position of a picture displayed by the 
picture outputting portion 23 with the control information 

5 inputting portion 24 ; it detects the position as a consid- 
ered point that the user is considering, places the coor- 
dinates of the considered point in the control informa- 
tion, and outputs the resultant control information. Alter- 
natively, the control information inputting portion 24 is 

10 composed of for example a video camera. When the us- 
er photographs and recognizes a picture, the control in- 
formation inputting portion 24 detects a point that the 
user is considering as a considered point, places the co- 
ordinates of the considered point in the control informa- 

15 tion, and outputs the resultant control information. The 
control information inputting portion 24 may be struc- 
tured so that the user can directly input the spatial res- 
olution and the temporal resolution of a picture dis- 
played by the picture outputting portion 23 as control 

20 information. 

[0109] When the control information transmitting por- 
tion 25 receives the control information from the control 
information inputting portion 24, the control information 
transmitting portion 25 transmits the control information 

25 to the transmitting apparatus 1 through the transmission 
path 3. 

[0110] Next, with reference to a flow chart shown in 
Fig. 6, the process of the receiving apparatus 2 shown 
in Fig. 5 will be described in brief. 
30 [0111] In the receiving apparatus 2, the receiving 
process portion 21 receives multiplexed data from the 
transmitting apparatus 1 through the transmission path 
3. At step S11, the receiving process portion 21 per- 
forms a receiving process for separating the multiplexed 
35 . data into a background picture, an object, and additional 
information. The receiving process portion 21 supplies 
the background picture, the object, and the additional 
information as the result of the receiving process to the 
combining process portion 22. At step S12, the combin- 
40 ing process portion 22 combines the background pic- 
ture, the object, and the additional information to the 
original picture and supplies the combined picture to the 
picture outputting portion 23. The picture outputting por- 
tion 23 displays the combined picture. Thereafter, the 
45 flow returns to step S1 1 . Thereafter, the receiving appa- 
ratus repeats the similar process. 
[0112] In the receiving apparatus 2, when the user 
designates a considered point of a picture displayed by 
the picture outputting portion 23 with the control infor- 
50 mation inputting portion 24, the coordinates of the con- 
sidered point are placed in the control information. The 
resultant control information is supplied to the control 
information transmitting portion 25. The control informa- 
tion transmitting portion 25 transmits the control infor- 
ms mation to the transmitting apparatus 1 through the trans- 
mission path 3. When the transmitting apparatus 1 re- 
ceives the control information, as was described above, 
the transmitting process portion 16 controls the trans- 
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mission of the background picture, the object, and the 
additional information so that the spatial resolution and 
the temporal resolution of the picture displayed by the 
receiving apparatus 2 can be changed while a predeter- 
mined condition is satisfied. Thus, thereafter, since the 
background picture, the object, and the additional infor- 
mation that have been controlled are transmitted from 
the transmitting apparatus 1 to the receiving apparatus 
2, a picture of which the spatial resolution and the tem- 
poral resolution have been changed while a predeter- 
mined condition has been satisfied is displayed by the 
receiving apparatus 2. 

[0113] Next, Fig. 7 shows an example of the structure 
of the transmitting process portion 1 6 of the transmitting 
apparatus 1 shown in Fig. 3. 

[01 14] A background picture, an object, and addition- 
al information are supplied from the pre-process portion 
12 (see Fig. 3) to the encoding portion 31 and the con- 
trolling portion 35. The encoding portion 31 encodes the 
background picture, the object, and the additional infor- 
mation and supplies the resultant encoded data to an 
MUX (multiplexer) 32. The MUX32 selects encoded da- 
ta of the background picture, the object, and the addi- 
tional information received from the encoding portion 31 
under the control of the controlling portion 35 and sup- 
plies the selected encoded data as multiplexed data to 
a transmitting portion 33. The transmitting portion 33 
modulates the multiplexed data received from the MUX 
32 and transmits the modulated data to the receiving 
apparatus 2 through the transmission path 3. A data 
amount calculating portion 34 monitors the multiplexed 
data that the MUX 32 outputs to the transmitting portion 
33, calculates the data rate of the multiplexed data, and 
supplies the calculated data rate to the controlling por- 
tion 35.' 

[0115] The controlling portion 35 controls the output 
of the multiplexed data of the MUX 32 so that the data 
rate of the data amount calculating portion 34 does not 
'exceed the transmission rate of the transmission path 
3. In addition, the controlling portion 35 receives the con- 
trol information transmitted from the receiving apparatus 
2 through the transmission path 3 and controls the se- 
lection of the encoded data of the MUX 32 correspond- 
ing to the control information. 

[0116] Fig. 8 shows an example of the structure of the 
encoding portion 31 shown in Fig. 7. 
[0117] A background picture is supplied to a differ- 
ence calculating portion 41 B. The difference calculating 
portion 41 subtracts a background picture of the imme- 
diately preceding frame that has been processed from 
a background picture of the current frame that is proc- 
essed (hereinafter, this frame may be referred to as con- 
sidered frame) and supplies difference data of the back- 
ground pictures as the subtracted result to a hierarchi- 
cally encoding portion 42B. The hierarchically encoding 
portion 42B hierarchically encodes the difference data 
of the background pictures received from the difference 
calculating portion 41 B and supplies the encoded result 



to a storing portion 43B. The storing portion 43B tempo- 
rarily stores the hierarchically encoded result received 
from the hierarchically encoding portion 42. The hierar- 
chically encoded result stored in the storing portion 43B 
5 is supplied as encoded data of background pictures to 
the MUX 32 (see Fig. 7). 

[01 18] The hierarchically encoded result stored in the 
storing portion 43B is supplied to a local decoder 44B. 
The local decoder 44B decodes the hierarchically en- 

10 coded result to the original background picture and sup- 
plies the decoded background picture to the difference 
calculating portion 41 B. The background picture decod- 
ed by the local decoder 44B is used by the difference 
calculating portion 41 B. The difference calculating por- 

15 tion 41 B uses the decoded background picture to obtain 
the difference data of a background picture of the next 
frame. 

[0119] Ah object is supplied to a difference calculating 
portion 41 F. The difference calculating portion 41 F, a hi- 

20 erarchically encoding portion 42F, a storing portion 43F, 
and a local decoder 44 F perform the same processes 
as the above-described difference calculating portion 
41 B s the hierarchically encoding portion 42B, the storing 
portion 43B, and the local decoder 44B. Thus, in the 

25 same manner as the background picture, the object is 
hierarchically encoded and supplied to the MUX 32 (see 
Fig. 7). When there are a plurality of objects, the differ- 
ence calculating portion 41 F, the hierarchically encoding 
portion 42F, the storing portion 43F, and the local de- 

30 coder 44F hierarchically encode the plurality of objects 
in the above-described manner. 

[0120] Additional information is supplied to a VLC 
(Variable Length Coding) portion 45. The VLC portion 
45 encodes the additional information corresponding to 
35 variable length code encoding method and supplies the 
encoded additional information to the MUX 32 (see Fig. 
7). 

[0121] Next, with reference to Fig. 9, hierarchical en- 
coding / decoding performed by the encoding portion 31 

40 shown in Fig. 8 will be described. 

[0122] Now, it is assumed that the average value of 
four pixels of 2x2pixels (horizontal x vertical) of a lower 
hierarchy is treated as a pixel (pixel value) of a higher 
hierarchy and that in three hierarchical levels, a picture 

45 is encoded. In this case, as a picture in the lowest hier- 
archical level, considering 8x8 pixels as shown in Fig. 
9 (A), the average value mO of four pixels hOO, h01 , h02, 
and h03 of an upper left portion of 2 x 2 pixels is calcu- 
lated. The average value mO is treated as one pixel of 

50 an upper left portion of the second hierarchical level. 
The average value ml of four pixels h10, h11 , h12, and 
h13 of an upper right portion of the lowest hierarchical 
level is calculated. The average value m2 of four pixels 
h20, h21 , h22, and h23 of a lower left portion of the iow- 

55 est hierarchical level is calculated. The average value 
m3 of four pixels h30, h31, h32, and h33 of a lower right 
portion of the lowest hierarchical level is calculated. The 
average values ml, m2, and m3 are treated as pixels at 
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an upper right portion, a lower left portion, and a lower 
right portion of the second hierarchical level, respective- 
ly. In addition, the average value q of four pixels mO, ml , 
m2, and m3 of 2 x 2 pixels of the second hierarchical 
level is calculated. The average value q is treated as a 5 
pixel of the third hierarchical level that is the highest hi- 
erarchical level. 

[01 23] According to the above-described hierarchical 
encoding, the spatial resolution of a picture in the high- 
est hierarchical level is the lowest. The spatial resolution 10 
of a picture is inversely proportional to the hierarchical 
level. In other words, the spatial resolution of a picture 
in the lowest hierarchical level is the highest. 
[0124] The data amount in the case that all the pixels 
hOO to h03, h10 to h13 : h20 to h23, h30 to h33, mO to 15 
m3, and q are transmitted is larger than the data amount 
in the case that only a picture of the lowest hierarchical 
level is transmitted by the pixels mO to m3 and q of the 
higher hierarchical levels. 

[0125] Thus, as shown in Fig. 9 (B), the pixel q of the 20 
third hierarchical level is transmitted instead of for ex- 
ample the pixel m3 at a lower right position of the pixels 
mO to m3 of the second hierarchical level. 
[0126] In addition, as shown in Fig. 9 (C), the pixel mO 
of the second hierarchical level is transmitted instead of 25 
the pixel h03 at a lower right position of the lowest hier- 
archical level. Likewise, the remaining pixels ml; m2, 
and q of the second hierarchical level are transmitted 
instead of the pixels h13, h23, and h33 of the first hier- 
archical level. Although the pixel q is not a pixel of the 30 
second hierarchical level, it is transmitted instead of the 
pixel m3 directly obtained from the pixels h30 to h33. 
Thus, the pixel q is transmitted instead of the pixel m3 
instead of the pixel h33. 

[0127] Thus, as shown in Fig. 9 (C), the number of 35 
pixels that are transmitted becomes 4x4 pixels = 16 
pixels. Thus, the number of pixels that are transmitted 
is the same as the case that only pixels of the lowest 
hierarchical level shown in Fig. 9 (A) are transmitted. 
Thus, in this case, the amount of data that is transmitted *o 
can be prevented from increasing. 
[0128] The pixel m3 transmitted instead of the pixel q 
and the pixels h03, h13, h23, and h33 transmitted in- 
stead of the pixels mO to m3 can be decoded in the fol- 
lowing manner. 45 
[0129] in other words, since q is the average value of 
mO to m3, the expression q = (mO + ml + m2 +■ m3) / 4 
is satisfied. Thus, the pixel m3 of the second h ierarch ical 
level can be obtained (decoded) with the pixel q of the 
third hierarchical level and the pixels mO to m2 of the so 
second hierarchical level corresponding to the expres- 
sion m3 = 4 x q - (mO + ml + m2). 
[0130] In addition, since mO is the average value of 
hOO to h03, the expression mO = (hOO 4- h01 + h02 + 
h03) / 4 is satisfied. Thus, corresponding to the expres- 55 
sion h03 = 4 x mO - (hOO + h01 + h02), the pixel h03 of 
the first hierarchical level can be obtained with the pixel . 
mO of the second hierarchical level and the pixels hOO 



to h02 of the first hierarchical level. In the same manner, 
hi 3, h23, and h33 can be obtained, 
[0131] Thus, a pixel that is not transmitted in a partic- 
ular hierarchical level can be decoded with pixels that 
are transmitted in the particular hierarchical level and a 
pixel of the immediately higher hierarchical level, 
[0132] Next, with reference to a flow chart shown in 
Fig. 10, the transmitting process of the transmitting proc- 
ess portion 16 shown in Fig. 7 will be described. 
[0133] At step S21 , in the transmitting- process portion 
16, the controlling portion 35 determines whether or not 
a control signal has been transmitted from the receiving 
apparatus 2. When the determined result at step S21 
represents that the control signal has not been transmit- 
ted from the receiving apparatus 2 (namely, the control- 
ling portion 35 has not received the control signal), the 
flow advances to step S22. The controlling portion 35 
controls the MUX 32 to select encoded data of a back- 
ground picture, an object, and additional information 
and multiplex them so that the receiving apparatus 2 can 
display a picture with a regular temporal resolution (for 
example, a default temporal resolution). 
[0134] When the regular temporal resolution is for ex- 
ample 30 frames / sec, the MUX 32 selects encoded 
data of a background picture, an object, and additional 
information, multiplex the encoded data, and outputs the 
multiplexed data so that the picture is displayed with the 
maximum spatial resolution of which the multiplexed da- 
ta is transmitted at the transmission rate of the transmis- 
sion path 3. 

[0135] In reality, for example, in the case that hierar- 
chical encoding is performed in three hierarchical levels, 
when data of only the third hierarchical level is transmit- 
ted at the transmission rate of the transmission path 3, 
encoded data of a background picture, an object, and 
additional information is selected so that a picture of the 
third hierarchical level is displayed. Thus, in this case, 
the receiving apparatus 2 displays a picture with a spa- 
tial resolution that is 1/4 times (in the horizontal direction 
and the vertical direction each) lower than that of the 
original picture (of the first hierarchical level) and a tem- 
poral resolution of 30 frames/sec. 
[0136] Thereafter, the flow advances to step S23. At 
step S23, the transmitting portion 33 transmits the mul- 
tiplexed data that is output from the MUX 32 through the 
transmission path 3. Thereafter, the flow returns to step 
S21 . 

[0137] When the determined result at step 521 repre- 
sents that a control signal has been transmitted from the 
receiving apparatus 2 (namely, the controlling portion 35 
has received the control signal), the flow advances to 
step S24. The controlling portion 35 operates the control 
information inputting portion 24 (see Fig. 5) correspond- 
ing to the control signal and thereby recognizes a des- 
ignated considered point. Thereafter, the flow advances 
to step S25. 

[0138] At step S25, the controlling portion 35 desig- 
nates an area such as a rectangle having a predeter- 
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mined size around a considered point (the rectangle has 
parallel sides in the horizontal direction and the vertical 
direction of a frame and has the considered point at the 
center of gravity thereof) as a priority range in which the 
spatial resolution is improved with priority and detects a 
background picture, an object, and additional informa- 
tion that compose the picture in the priority range. 
[0139] Thereafter, the flow advances to step S26. At 
step S26, the controlling portion 35 controls the MUX 32 
to select and multiplex encoded data of a background 
picture, an object, and additional information and multi- 
plex the selected encoded data so that the receiving ap- 
paratus 2 can display the picture in the priority range 
with a higher spatial rpsolution. 

[0140] In other words, when the controlling portion 35 
receives a contrcl signal from the receiving apparatus 
2, the controlling portion 35 controls the MUX 32 to im- 
prove the spatial resolution of the picture in the priority 
range at the sacrifice of the temporal resolution. 

\ [0141] Thus, the MUX 32 selects encoded data of a 
background picture, an object, and additional informa- 
tion necessary for displaying a picture of the third hier- 
archical level and a picture of the second hierarchical 
level with priority, multiplexes the selected encoded da- 
ta, and outputs the multiplexed data. 
[0142] In addition, at step S26, the controlling portion 
35 controls the MUX 32 to insert information of the po- 
sition of the priority range, the size thereof, and so forth 
(hereinafter may be referred to as high resolution infor- 
mation) into the additional information selected as mul- 
tiplexed data. The* eafter, the flow advances to step S23. 
[0143] At step S23, as was described above, the 
transmitting portion 33 transmits the multiplexed data 
that is output from the MUX 32 through the transmission 
path 3. Thereafter, the flow returns to step S21 . 
[0144] For simplicity, assuming that encoded data of 
a background picture, an object, and additional informa- 
tion for displaying a picture of the third hierarchical level 
! is continuously selected for a picture of the third hierar- 

^ chical level as a -picture in the non-priority region at step 
S26, the data amount of the multiplexed data is larger 
than that in the case at step S22 by the data of the sec- 
ond hierarchical level for the picture in the priority range. 
According to the embodiment of the present invention, 
as was der cribed above, to display a picture at 30 
frames/second, data of only the third hierarchical level 
is transmitted at the transmission rate of the transmis- 
sion path 3. Thus, the multiplexed data obtained at step 
S26 cannot be transmitted so that the picture is dis- 
played at 30 frames/second. Thus, at step S23, in the 
extreme case, the multiplexed data obtained at step S26 
is transmitted with a temporal resolution of 0 frames/sec. 
[0145] Thus, in this case, for a picture in the priority 
range the receiving apparatus 2 displays a picture of 
which the spatial resolution in each of the horizontal di- 
rection and the vertical direction is 1/2 as low as that of 
the original picture (of the first hierarchical level). In oth- 
er words, the receiving apparatus 2 displays a picture 



(of the second hierarchical level) of which the spatial 
resolution in each of the horizontal direction and the ver- 
tical direction is twice as high as that of a picture of the 
third hierarchical level. However, the temporal resolu- 

5 tion is 0 frame / second. In other words, the receiving 
apparatus 2 displays a still picture. 
[0146] After data of the second hierarchical level is 
transmitted for a picture in the priority range, when the 
determined result at step S21 represents thatthe control 

10 signal has been transmitted again from the receiving ap- 
paratus 2 (namely, the user continues to operate the 
control information inputting portion 24 and designates 
the same considered point), the flow advances to step 
S24. At step S24, the controlling portion 35 recognizes 

*s the same considered point. Thereafter, the flow advanc- 
es to step S25. At step S25, the controlling portion 35 
designates the same priority range. Thereafter, the flow 
advances to step S26 . 

[01 47] At step S26, the controlling portion 35 controls 

20 the MUX 32 to select encoded data of a background pic- 
ture, an object, and additional information and multiplex- 
es the selected encoded data so that the receiving ap- 
paratus 2 can display the picture in the priority range 
with a higher spatial resolution. 

25 [0148] In this case, for a picture in the priority range, 
encoded data of a background picture, an object, and 
additional information of the third hierarchical level and 
the second hierarchical level is selected with priority. In 
this case, encoded data of a background picture, an ob- 

30 ject, and additional information of the first hierarchical 
level is selected with priority and output as multiplexed 
data. In addition, at step S26, as was described above, 
high resolution information is inserted into the additional 
information. Thereafter, the flow advances to step S23. 

35 At step S23, the transmitting portion 33 transmits the 
multiplexed data that is output from the MUX 32 through 
the transmission path 3. Thereafter, the flow returns to 
step S21 . 

[0149] Thus, in this case, for a picture in the priority 

40 range, the receiving apparatus 2 displays a picture with 
a spatial resolution that is the same as that of the original 
picture (of the first hierarchical level). In other words, the 
receiving apparatus 2 displays a picture (of the first hi- 
erarchical level) with a spatial resolution (in each of the 

45 horizontal direction and the vertical direction) that is four 
times higher than that of the picture of the third hierar- 
chical level. However, the temporal resolution is for ex- 
ample 0 frames/second. In other words, the receiving 
apparatus 2 displays a still picture. 

so [0150] Thus, when the user continues to operate the 
control information inputting portion 24 and designate 
the same considered point, for a picture in the priority 
range that contains the considered point, data for im- 
proving a spatial resolution is transmitted with priority at 

55 the sacrifice of the temporal resolution of the picture. 
Thus, although the temporal resolution of the picture de- 
teriorates, the spatial resolution of the picture in the pri- 
ority range that contains the considered point gradually 
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improves. Thus, the picture in the priority range is more 
clearly displayed. In other words, the picture of the por- 
tion that the user is considering is more clearly dis- 
played. 

[0151] Thus, the transmission of picture data is con- 
trolled so that the spatial resolution and the temporal 
resolution of a picture in a particular priority range that 
contains a considered point are changed in the range of 
a resolution of picture data transmitted at the transmis- 
sion rate of the transmission path 3, In other words, the 
transmission of picture data is controlled so that the spa- 
tial resolution of a picture in the priority range improves 
and the temporal resolution of the picture deteriorates. 
Thus, the improved spatial resolution and the deterio- 
rated temporal resolution is obtained with picture data 
transmitted at the transmission rate of the transmission 
path 3. As a result, at a limited transmission rate, the 
spatial resolution of a picture displayed by the receiving 
apparatus 2 can be more improved. 
[0152] Next, Fig. 11 shows an example of the struc- 
ture of the receiving process portion 21 shown in Fig. 5. 
[01 53] Multiplexed data is received by a receiving por- 
tion 51 through the transmission path 3. After the multi- 
plexed data is demodulated, it is supplied to a DMUX 
(demultiplexer) 52. The DMUX 52 separates the multi- 
plexed data received from the receiving portion 51 into 
encoded data of a background picture, an object, and 
additional information and supplies the separated data 
to a decoding portion 53. The decoding portion 53 de- 
codes the encoded data of the background picture, the 
object, and the additional information into their original 
data and outputs them to the combining process portion 
22 (see Fig. 5). 

[0154] In other words, Fig. 12 shows an example of 
the structure of the decoding portion 53 shown in Fig. 1 1 . 
[0155] Hierarchically encoded difference data as en- 
coded data of a background picture is supplied to an 
adding device 61 B. In addition, a background picture of 
the immediately preceding frame that has been decod- 
ed and stored in a storing portion 62B is supplied to the 
adding device 61 B. The adding device 61 B adds the dif- 
ference data of background pictures and the back- 
ground picture of the immediately preceding frame re- 
ceived from the storing portion 62B so as to decode a 
background picture of a desired hierarchical level. The 
decoded background picture is supplied to the storing 
portion 62B. The storing portion 62B stores the back- 
ground picture received from the adding device 61 B. 
The decoded background picture is supplied to the add- 
ing device 61 B and the combining process portion 22 
(see Fig. 5). 

[0156] Hierarchically encoded difference data as en- 
coded data of an object is supplied to an adding device 
61 F. The adding device 61 F and a storing portion 62F 
perform the similar processes to those of the adding de- 
vice 61 B and the storing portion 62B. Thus, in the same 
manner as the background picture, the difference data 
of objects is decoded to an object of a desired hierar- 



chical level and then supplied to the combining process 
portion 22 (see Fig. 5). When there are a plurality of ob- 
jects, the adding device 61 F and the storing portion 62F 
decode (hierarchically decode) the difference data of 
5 the plurality of objects. 

[0157] Additional information that has been encoded 
with variable length code as encoded data of additional 
information is supplied to an inverse VLC portion 63. 
The inverse VLC portion 63 decodes the additional in- 
fo formation with variable length code. Thus, the original 
additional information is obtained and supplied to the 
combining process portion 22. 

[0158] The local decoder 44B shown in Fig. 8 is struc- 
tured as the adding device 61 B and the storing portion 

15 62B. Likewise, the local decoder 44F is structured as 
the adding device 61 F and the storing portion 62F. 
[0159] Next, Fig.. 13 shows an example of the struc- 
ture of the combining process portion 22 shown in Fig. 5. 
[0160] A background picture that is output from the 

20 decoding portion 53 (see Fig. 1) is supplied to a back- 
ground picture writing portion 71 . An objectthat is output 
from the decoding portion 53 is supplied to an object 
writing portion 72. Additional information that is output 
from the decoding portion 53 is supplied to the back- 

25 ground picture writing portion 71 , the object writing por- 
tion 72, and a combining portion 77. 
[0161] The background picture writing portion 71 suc- 
cessively writes background pictures to a background 
picture memory 73. When a picture is photographed 

so while the video camera is panned ortilted, a background 
picture is moved. At that point, the background picture 
writing portion 71 aligns the background picture and 
then writes the aligned background picture to the back- 
ground picture memory 73. Thus, the background pic- 
as ture memory 73 can store a picture that is spatially wider 
than a picture of one frame. The background picture is 
aligned corresponding to a background picture moving 
vector contained in the additional information. 
[0162] When the background picture writing portion 

40 71 writes a background picture with a high spatial reso- 
lution to the background picture memory 73, the back- 
ground picture writing portion 71 changes the value of 
a background picture flag stored at an address of a 
background picture flag memory 74 corresponding to 

45 the pixel of the background picture from 0 to 1. When 
the background picture writing portion 71 writes a back- 
ground picture to the background picture memory 73, 
the background picture writing portion 71 references the 
background picture flag memory 74. The background 

so picture writing portion 71 does not write a background 
picture with a low spatial resolution to a pixel whose 
background picture flag is 1 that represents that the pixel 
storing a background picture with a high spatial resolu- 
tion. Thus, basically, whenever a background picture is 

55 supplied to the background picture writing portion 71 , it 
is written to the background picture memory 73. How- 
ever, the background picture writing portion 71 does hot 
write a background picture with a low spatial resolution 
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to a pixel that stores a background picture with a high 
spatial resolution. As a result, whenever a background 
picture with a high spatial resolution is supplied to the 
background picture writing portion 71, the range of a 
high spatial resolution becomes wide in the background 
picture memory 73. 

[0163] The object writing portion 72 successively 
writes supplied objects to an object memory 75. When 
there are a plurality of objects, the object writing portion 
72 writes each object to the object memory 75. When 
the object writing portion 72 writes the same object (as- 
signed the same label) to the object memory 75, the ob- 
ject writing portion 72 substitutes the old object with the 
new object (that is newly supplied to the object writing 
portion 72). 

[0164] In addition, when the object writing portion 72 
writes an object with a high spatial resolution to the ob- 
ject memory 75, the object writing portion 72 changes a 
background picture flag stored at an address of an ob- 

v | ject flag memory 76 corresponding to the pixel of the 
object from 0 to 1 . When the object writing portion 72 
wriies an object to the object memory 75, the object writ- 
ing portion 72 references the object flag memory 76. The 
object writing portion 72 does not write an object with a 
low spatial resolution to the object memory 75 that 
stores an object with a high spatial resolution (object flag 
= 1 ). Thus, in the same manner as the background pier 
ture memory 73, whenever an object is supplied to the 
object writing portion 72, the object is written to the ob- 
ject writing portion 72. However, the object writing por- . 
tion 72 does not write an object with a low spatial reso- 
lution to a pixel that stores an object with a high spatial 
resolution. As a result, whenever an object with a high 
spatial resolution is supplied to the object writing portion 
72. the number of objects with a high spatial resolution 
increases in the object memory 75. 
[0165] The background picture memory 73 stores a 
background picture supplied from the background pic- 
ture writing portion 71. The background picture flag 

y memory 74 stores the above-described one-bit back- 
ground picture flag at an address of the background pic- 
ture memory 73. The background picture flag memory 
74 represents whether or not a background picture with 
a high spatial resolution has been stored at an address 
of the background picture memory 73. The object mem- 
ory 75 is composed of at least one memory portion that 
stores each object supplied from the object writing por- 
tion 72. The object flag memory 76 stores the above- 
described one-bit object flag that represents whether or 
not an object with a high spatial resolution has been 
stored in the object memory 75. 

[0166] In this case, for simplicity, the background pic- 
ture flag and the object flag are one-bit flags. Alterna- 
tively, the background picture flag and the object flag 
may be multi-bit flags. In that case, the background pic- 
ture flag and the object flag can represent resolutions in 
more levels. In other words, a one-bit flag represents 
only two levels that are a high resolution and a low res- 



olution. However, a multi-bit flag represents resolutions 
in more levels. 

[0167] The combining portion 77 reads a background 
picture of the current frame (this frame may be referred 

s to as considered frame) stored in the background pic- 
ture memory 73 corresponding to a background picture 
moving vector contained in the additional information. 
In addition, the combining portion 77 combines the 
background picture and an object stored in the object 

to memory 75 corresponding to an object moving vector 
contained in the additional information. As a result, the 
combining portion 77 generates a picture of the consid- 
ered frame and supplies the picture to a display memory 
78. 

is [0168] In addition, when the combining portion 77 re- 
ceives control information from the control information 
inputting portion 24 (see Fig. 5), the combining portion 
77 reads an object at the considered point contained in 
the control information from the object memory 75 and 

20 supplies the object to a sub window memory 79. 

[0169] The display memory 78 functions as a so- 
called VRAM (Video Read Only Memory). The display 
memory 78 temporarily stores the picture of the consid- 
ered frame supplied from the combining portion 77. 

25 [01 70] The sub window memory 79 temporarily stores 
an object supplied from the combining portion 77. 
[0171] A superimposing portion 80 reads the stored 
content of the display memory 78 and supplies the 
stored content to the picture outputting portion 23 (see 

30 Fig. 5). The picture outputting portion 23 displays the 
stored content. When necessary, the superimposing 
portion 80 opens a sub window (that will be described 
later) on the picture outputting portion 23, reads the 
stored content of the sub window memory 79, and dis- 

35 plays the content on the sub window. 

[0172] Next, with reference to Fig. 14, the process 
(combining process) performed by the combining proc- 
ess portion 22 shown in Fig. 13 will be described. 
[0173] First of all, at step S31 , the background picture 

40 writing portion 71 or the object writing portion 72 writes 
a background picture or an object that is supplied from 
the decoding portion 53 (see Fig. 12) to the relevant 
memory corresponding to the background picture flag 
stored in the background picture flag memory 74 or the 

45 object flag stored in the object flag memory 75, respec- 
tively, in the above-described manner. 
[0174] In other words, the background picture writing 
portion 71 references the background picture flag mem- 
ory 74 and writes the supplied background picture to an 

50 address of the background picture memory 73 when the 
background picture flag of a pixel corresponding to the 
address is 0. In contrast, if the spatial resolution of the 
supplied background picture is high, the background 
picture writing portion 71 writes the background picture 

55 to an address of the background picture memory 73 
when the background picture flag of a pixel correspond- 
ing to the address is 1 . 

[0175] Likewise, when the object flag of the object 
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memory 75 is 0, the object writing portion 72 writes the 
supplied object to the object memory 75. When the ob- 
ject flag of the object memory 75 is 1 , if the spatial res- 
olution of the supplied object is high, the object writing 
portion 72 writes the supplied object to the object mem- 
ory 75. 

[0176] When the background picture writing portion 
71 writes a background picture to a particular address 
of the background picture memory 73, if the address has 
stored a background picture, the background picture 
writing portion 71 overwrites the background picture to 
the address. This operation applies to the object mem- 
ory 75. 

[0177] Thereafter, (he flow advances to step S32. At 
step S32, the background picture writing portion 71 and 
the object writing portion 72 determine whether or not 
the additional information contains high resolution infor- 
mation. When the determined result at step S32 repre- 
sents that the additional information contains the high 
resolution information (namely, the user has operated 
the control information inputting portion 24 (see Fig. 5), 
the resultant control information has been transmitted 
to the transmitting apparatus 1 , and then the transmit- 
ting apparatus 1 has transmitted a background picture 
and an object with a high spatial resolution for a picture 
in the priority range), the flow advances to step 33. At 
step S33, the background picture writing portion 71 or 
the object writing portion 72 changes the background 
picture flag of the background picture flag memory 74 
or the object flag of the object flag memory 76 to 1 . 
[0178] In othet words, when the transmitting appara- 
tus 1 transmits a oackground picture and an object with 
a high spatial resolution for a picture in the priority range, 
at step S31, the background picture writing portion 71 
and the object writing portion 72 write the background 
picture and object with the high spatial resolution to the 
background picture memory 73 and the object memory 
75, respectively. Thus, at step S33, the background pic- 
ture flag and the object flag for a pixel that composes 
the background picture and the object with the high spa- 
tial resolution are changed to 1 . 

[01 79] Thereafter, the flow advances to step S34. The 
combining portion 77 reads an object in the priority 
range from the object memory 75 and writes the object 
to the sub window memory 79. 

[01 80] in other words, when the determined result at 
step S32 represents that the additional information con- 
tains high resolution information, the user has operated 
the control information inputting portion 24 (see Fig. 5), 
the control information has been transmitted to the 
transmitting apparatus 1 , and then the transmitting ap- 
paratus 1 has transmitted a background picture and an 
object with a high spatial resolution for a picture in the 
prior ity range. The control information that is transmitted 
to the transmitting apparatus 1 is also supplied to the 
combining portion 77. Thus, when the combining portion 
77 receives the control information, at step S34, the 
combining portion 77 recognizes the priority range cor- 



responding to the coordinates of the considered point 
contained in the control information, reads an.object with 
a high spatial resolution in the priority range that has 
been transmitted from the transmitting apparatus 1 from 
the object memory 75, and writes the object to the sub 
window memory 79. ' 

[0181] Thereafter, the flow advances to step S35. At 
step S35, the combining portion 77 reads a background 
picture of the current frame that is displayed (this frame 
is referred to as considered frame) from the background 
picture memory 73 corresponding to a background pic- 
ture moving vector contained in the additional informa- 
tion. In addition, the combining portion 77 reads an ob- 
ject that is displayed on the considered frame from the 
object memory 75. The combining portion 77 combines 
the background picture of the considered frame and the 
object that Is read from the object memory 75 corre- 
sponding to the object moving vector contained in the 
additional information and writes the combined picture 
of the considered frame to the display memory 78. In 
other words, the combining portion 77 writes a back- 
ground picture to for example the display memory 78 
and then overwrites an object to the display memory 78. 
As a result, the combining portion 77 writes a picture of 
25 which a background picture and an object are combined 
A as a picture of the considered frame to the display mem- 
ory 78. 

[0182] In such a manner, a picture of the considered 
frame written to the display memory 78 and an object 
30 written to the sub window memory 79 are supplied to 
the picture outputting portion 23 (see Fig. 5). The picture 
outputting portion 23 displays the supplied picture and 
object. 

[01 83] In the transmitting apparatus 1 , a so-called soft 
35 key may be contained in for example additional informa- 
tion. In this case, the combining portion 77 combines an 
object and a background picture using the soft key. 
[0184] In contrast, when the determined result at step 
S32 represents that the additional information does not 
40 contain high resolution information (namely, the user 
has not operated the control information inputting por- 
tion 24 (see Fig. 5)), the flow advances to step S35, skip- 
ping steps 33 and 34. As was described above, the com- 
bining portion 77 reads a background picture of thecon- 
45 sidered frame from the background picture memory 73 . 
In addition, the combining portion 77 reads a required 
object from the object memory 75 and combines the 
background picture of the considered frame and the ob- 
ject that is read from the object memory 75 correspond- 
50 ing to the additional information. As a result, a picture 
of the considered frame is generated and written to the 
display memory 78. Thereafter, the flow returns to step 
S31. Thereafter, the combining process portion 22 re- 
peats the similar process. 
55 [0185] In the above-described combining process, 
when the user does not operate the control information 
inputting portion 24 (see Fig. 5) (namely, the control in- 
formation inputting portion 24 that is composed of a 
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pointing device such as a mouse is not dragged (or 
clicked)), as shown in Fig. 15 (A), a picture with a low 
spatial resolution is displayed with a default temporal 
resolution on the picture outputting portion 23 (see Fig. 
5). In Fig. 15 (A), an object with a low spatial resolution 
is being moved on a background picture with a low spa- 
tial resolution. 

[0186] When the user operates the control informa- 
tion inputting portion 24 (see Fig. 5), moves the cursor 
to an object, and drags the cursor at the position of the 
object, as was described above, control information is 
transmitted to the transmitting apparatus 1 . The trans- 
mitting apparatus 1 transmits data necessary for dis- 
playing a picture with a high spatial resolution at the sac- 
rifice of a temporal resolution for a picture in the priority 
range. As a result, as shown in Fig. 15 (B), although the 
temporal resolution Is for example 0 frames/second, a 
picture of which the spatial resolution of an object and 
;<a background picture in the priority range around the 
) dragged position is gradually improved is displayed on 
the picture outputting portion 23 (see Fig 5). In other 
words, corresponding to the dragging time (or the 
number of clicks), the spatial resolution of a picture in 
the priority range is gradually improved. 
[0187] In this case, on the picture outputting portion 

23 (see Fig. 5), as shown in Fig! 15 (B), a sub window 
is opened. An object in the priority range around the 
dragged position is displayed in such a manner that the 
spatial resolution of the object is gradually improved. 
[0188] Thereafter, when the user stops the dragging 
operation with the control information inputting portion 

24 (see Fig. 5), as was described above, at step S35, 
the combining portion 77 reads a background picture of 
the considered frame from the background picture 
memory 73, reads an object from the object memory 75, 
combines the background picture of the considered 
frame and the object that is read from the object memory 

. 75 corresponding to additional information, and writes 
'the combined picture to the display memory 78. As was 
y described above, since the object with a high spatial res- 
olution as a result of the dragging operation is stored in 
the object memory 75, the picture outputting portion 23 
(see Fig. 5) displays a picture of which an object with a 
high spatial resolution as a result of the dragging oper- 
ation is pasted at a relevant position of the considered 
frame with a regular temporal resolution as shown in Fig. 
15(G). 

[0189] Thereafter, the picture outputting portion 23 
(see Fig. 5) displays a picture of which an object with a 
high spatial resolution is being moved corresponding to 
the additional information with a regular temporal reso- 
lution. 

[01 90] Thus, when the user performs the dragging op- 
eration at the position of an object that he or she wants 
to see in detail, he or she can see the object with a high 
- spatial resolution. In other words, the user can see an 
object in detail. 

[0191] When the user performs the dragging opera- 



tion, a background picture with a high spatial resolution 
in the priority range is stored to the background picture 
memory 73. Thus, even if the user stops the dragging 
operation, the background picture with the high spatial 

5 resolution stored in the background picture memory 73 
is displayed. Thus, when the user performs the dragging 
operation, since the spatial resolution of a background 
. picture in the priority range including the dragged posi- 
tion is improved, whenever the user performs the drag- 

10 ging operation at each position on the display screen of 
the picture outputting portion 23 (see Fig. 5), a back- 
ground picture with a high spatial resolution gradually 
widens in a mosaic pattern. Finally, the picture output- 
ting portion 23 displays all the background picture with 

15 a high spatial resolution. 

[0192] In addition, according to the embodiment, as 
was described above, since a background picture Is 
stored in the background picture memory 73, the trans- 
mitting apparatus 1 does not need to transmit a back- 

20 ground picture with a low spatial resolution that has 
been already transmitted. Thus, the transmission band 
(transmission rate) for the background picture can be 
assigned to an object and a background with a higher 
spatial resolution. 

25 [0193] In the above-described case, an object with a 
high spatial resolution as a result of the dragging oper- 
ation is stored to the object memory 75. After the drag- 
ging operation is stopped, the object with the high spatial 
resolution is pasted to a background picture. Thus, the 

30 spatial resolution of an object displayed by the receiving 
apparatus 2 becomes high. However, the state of an ob- 
ject photographed by the transmitting apparatus 1 is not 
affected to an object displayed by the receiving appara- 
tus 2. 

35 [0194] To solve such a problem, afterthe dragging op- 
eration is stopped, an object with a high spatial resolu- 
tion stored in the object memory 75 can be substituted 
with an object stored in the storing portion 62F of the 
decoding portion 53 (see Fig. 12), ignoring the object 

40 flag. In other words, objects that are transmitted from 
the transmitting apparatus 1 are successively stored to 
the storing portion. 62F of the decoding portion 53 (see 
Fig. 1 2). Thus, when the objects are written to the object 
memory 75, the objects of a picture displayed on the 

45 picture outputtlhg portion 23 are affected by the state of 
the objects photographed by the transmitting apparatus 
1 (however, the spatial resolution of the objects dis- 
played is low). 

[0195] The MUX 32 shown in Fig. 7 that composes 
50 the transmitting process portion 1 6 of the transmitting 
apparatus 1 places the presence / absence of high res- 
olution information, a frame rate (temporal resolution) 
and a spatial resolution in a priority range corresponding 
to the high resolution information, and a frame rate and 
55 a spatial resolution in the non-priority range to a header 
or the like of multiplexed data under the control of the 
controlling portion 35. The receiving apparatus 2 recog- 
nizes the presence / absence of the high resolution in- 
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formation, the frame rate and spatial resolution in the 
priority range, and the frame rate and spatial resolution 
in the non-priority range corresponding to information 
placed in the header (hereinafter, this information is re- 
ferred to as header information). 
[0196] In the transmitting apparatus 1 , the header in- 
formation may contain for example (the coordinates) of 
a considered point contained in control information 
transmitted from the receiving apparatus 2. in this case, 
the combining portion 77 shown in Fig. 1 3 can recognize 
an object at the position of a considered point corre- 
sponding to the header information transmitted from the 
transmitting apparatus 1 . In other words, in the above- 
described case, the combining portion 77 recognizes an 
object at the position of a considered point contained in 
the control information corresponding to the control in- 
formation supplied from the control information inputting 
portion 24 (see Fig. 5), reads the object from the object 
memory 75, and supplies the object to the sub window 
memory 79. However, when the header information conr 
tatns a considered point, the combining portion 77 rec- 
ognizes an object at the position of the considered point 
corresponding to the header information. In this case, 
in Fig. 5, the control information inputting portion 24 
does not need to supply the control information to the 
receiving apparatus 2. 

[01 97] Next, with reference to Fig. 1 6, the relation be- 
tween a spatial resolution and a temporal resolution of 
a picture transmitted from the transmitting apparatus 1 
to the receiving apparatus 2 through the transmission 
path 3 will be described. 

[0198] Now, it is assumed that the transmission rate 
of the transmission path 3 is R [bps] and that a picture 
composed of a background picture and three objects #1 
to #3 is transmitted. For simplicity, in this example, ad- 
ditional information is not considered. In addition, it is 
assumed that to display the background picture and the 
objects #1 to #3 with a particular spatial resolution, each 
of them requires the same data amount. 
[0199] In this case, when the user does not perform 
the dragging operation, in the transmitting apparatus 1 , 
as shown in Fig. 1 6 (A), each of the background picture 
and the objects #1 to #3 is transmitted at R / 4 [bps] of 
which the transmission rate R is divided by 4. When the 
regular temporal resolution is 1/T frames/second, the 
transmitting apparatus 1 can transmit data of one frame 
of each or the background picture and the objects #1 to 
UZ in at most Tseconds. Thus, in this case, the receiving 
apparatus 2 displays the background picture and the ob- 
jects #1 to #3 with a spatial resolution obtained with data 
of T x R/4 bits per frame. 

[0200] When the user perform the dragging operation 
at the position of for example the object #1 , as shown 
in Fig. 16 (A), the transmitting apparatus 1 stops trans- 
mitting the background picture and the objects #2 and 
#3 and transmits only the object #1 at all the transmis- 
sion rate R of the transmission path 3. Thereafter, at time 
t2 of which time period 4T has elapsed from time t1 , 



when the user stops the dragging operation, the trans- 
mitting apparatus 1 transmits the background picture 
and the objects #1 to #3 at the transmission rate R/4 
each. 

5 [0201] Thus, while the userisperformingthe dragging 
operation, the transmitting apparatus 1 transmits data 
of 4T x R bits for the object #1. Thus, assuming that 
while the user is performing the dragging operation, the 
temporal resolution is 0 frames/second, the receiving 

10 apparatus 2 displays the object #1 with a spatial reso- 
lution obtained with data of 4T x R bits per frame. In 
other words, when the horizontal spatiai resolution and 
the vertical spatial resolution are improved for the same 
amount, in the receiving apparatus 2, although thetem- 

*5 poral resolution becomes 0 frames/second, the receiv- 
ing apparatus 2 displays the object #1 with a spatial res- 
olution in each of the horizontal direction and the vertical 
direction that is four times (= (4Tx R/(Tx R/4 bits)) 
higher than that in the case that the user does not per- 

20 form the dragging operation. 

[0202] In other words, the spatial resolution can be 
more improved at the sacrifice of the temporal resolu- 
tion. In addition, in comparison with the case that the 
temporal resolution is sacrificed, the spatial resolution 

25 of a portion that the user is considering can be quickly 
improved. 

[0203] In the above-described case, while the user is 
performing the dragging operation for the object #1 , all 
data of the background picture and the other objects #2 

30 and #3 is not transmitted. However, as shown in Fig. 16 
(B), a higher transmission rate can be assigned to the 
data of the object #1 , while a low transmission rate can 
be assigned to data of the background picture and the 
other objects #2 and #3. 

35 [0204] Alternatively, even if the dragging operation is 
performed, the transmission rate assigned to each of the 
background picture and the objects #1 to #3 may not be 
changed from R/4. In other words, in this case, since the 
spatial resolution is improved at the sacrifice of the tem- 

40 poral resolution, for example, although it takes a long 
time to display the data, the spatial resolution can be 
improved without need to change the assignment of the 
transmission rate. 

[0205] According to the embodiment, as was de- 
45 scribed above, an object with a high spatial resolution 
as a result of the dragging operation Is stored to the ob- 
ject memory 75 . After the dragging operation is stopped, 
the object with the high spatial resolution is pasted on a 
background picture. However, the position of the back- 
50 ground picture to which the object with the high spatial 
resolution is pasted depends on an object moving vector 
contained in the additional information about the object 
that is transmitted from the transmitting apparatus 1 . 
[0206] Thus, the receiving apparatus 2 should recog- 
55 nize what object of a particular frame corresponds to 
what object of an adjacent frame. Thus, when the object 
extracting portion 14 of the transmitting apparatus 1 
(see Fig. 3) extracts an object, the object extracting por- 
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tion 14 adds information that allows the receiving appa- 
ratus 2 to perform such an recognizing operation to the 
. additional information. 
[0207] Fig. 17 shows an example of the structure of 
the object extracting portion 14 shown in Fig. 3. 
[0208] A picture that is output from the picture input- 
ting portion 11 and a background picture that is output 
from the background picture extracting portion 13 are 
supplied to a subtracting portion 81 . The subtracting 
portion 18 subtracts the background picture that is out- 
put from the background picture extracting portion 13 
from the picture that is output from the picture inputting 
portion 11 and obtains a foreground picture as an object. 
The foreground picture obtained by the subtracting por- 
tion 81 is supplied to a frame memory 82 and an initial 
area dividing portion 83. 

[0209] The frame memory 82 temporarily stores the 
foreground picture supplied from the subtracting portion 
j81 . The initial area dividing portion 83 performs an initial 
\j area dividing process using the foreground picture of the 
current frame that is processed (this frame is referred to 
as considered frame) that is supplied from the subtract- 
ing portion 81 and the foreground picture of the imme- 
diately preceding frame stored in the frame memory 82 
(this foreground picture is referred to as foreground pic- 
ture of the preceding frame). 

[0210] In other words, the initial area dividing portion 
83 categorizes each pixel that composes the foreground 
picture of the considered frame as a class correspond- 
ing to the pixel value. In reality, when a pixel value is 
represented by RGB (Red, Green, and Blue), the initial 
area dividing portion 83 categorizes each pixel as a 
class depending on the relation between a vector com- 
posed of elements of R, G, and B values (this vector 
may be referred to as color vector) and a plurality of 
small areas in the RGB space. In addition , the initial area 
dividing portion 83 categorizes each pixel that compos- 
es the foreground picture of the immediately preceding 
■'frame stored in the frame memory 82 as a class in the 
); same manner. 

[021 1 ] When the considered frame is for example the 
n-th frame, the initial area dividing portion 83 divides the 
foreground picture of the n-th frame and the foreground 
picture of the (n-l)-th frame the immediately preceding 
frame into areas that are temporally or spatially adjacent 
areas and that are composed of pixels categorized as 
the same class. 

[021 2] In other words, for example, it is assumed that 
each pixel that composes the foreground picture of the 
n-th frame and each pixel that composes the foreground 
picture of the (n-1)-th frame are categorized as classes 
as shown in Fig. 18 (A). In Fig. 18 (also in Figs. 19 and 
20), a set of a character c and a numeral contained in a 
square represents the class of a pixel. 
[0213] In the case shown in Fig. 18 (A), when the fore- 
ground picture of the n-th frame and the foreground pic- 
ture of the (n-1)-th frame are divided into areas com- 
posed of pixels categorized as the same class, initial ar- 
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eas as denoted by dotted lines shown in Fig. 18 (B) are 
formed. 

[0214] The initial areas obtained by the initial area di- 
viding portion 83 are suppliedto an area merging portion 

5 84 shown in Fig. 17. 

[0215] The area merging portion 84 performs an area 
margining process for margining the initial areas sup- 
plied form the initial area dividing portion 83. 
[0216] In other words, the area merging portion 84 

10 reads object information about an object contained in 
the (n-1)-th frame from an object information memory 
88 and recognizes the position and the range of the ob- 
ject contained in the (n-1 )-th frame, in addition, the area 
merging portion 84 recognizes pixels that compose the 

15 object contained in the (n-1)-th frame and merges the 
initial areas that contain the pixels. 
[0217] In reality, for example, assuming that a partic- 
ular object Obj is contained in a range denoted by solid 
lines (a box) shown in Fig. 1 9 (A), the object Obj is com- 

20 posed of pixels categorized as classes c2, c3, c4, and 
c5. The initial areas containing these pixels are merged. 
Thus, in this example, initial areas composed of pixels 
of the classes c2, c3, c4, and c5 (hereinafter, the initial 
areas composed of pixels of the classes c#i are referred 

25 to as initial areas c#i) are merged as hatched areas 
shown in Fig. 19 (B). 

[0218] In addition, the area merging portion 84 calcu- 
lates the distance between the merged initial area (here- 
inafter may be referred to as merged area) and an initial 

30 area adjacent to the merged area. In this example, the 
distance between a merged area and an initial area ad- 
jacent thereto (hereinafter may be referred to as adja- 
cent initial area) may be the distance of the average val- 
ues of the pixel values (colors) of the pixels that com- 

35 pose two areas (the merged area and the adjacent initial 
area) (this distance may be referred to as distance in 
the RGB space) or the continuity of pixel values (colors) 
of pixels in the vicinity of the boundary of the two areas. 
[0219] When the distance between the merged area 

40 and the adjacent initial area is smaller than (or equal to) 
a predetermined threshold value, the area merging por- 
tion 84 merges the merged area and the adjacent initial 
area and generates a new merged area. The area merg- 
ing portion 84 calculates the above-described distance 

45 until there is no adjacent initial area that can be merged 
to the merged area and repeatedly merges the adjacent 
initial area and the merged area. 

[0220] Thus, with the merged area shown in Fig. 19 
(B), for example a merged area shown in Fig. 19 (C) is 

50 formed. In Fig. 19 (C), since the distance between the 
initial areas c7 and c8 is close, they are merged to the 
merged area shown in Fig. 1 9 (B). In contrast, since the 
distance between the initial areas cl and c6 is far, they 
are not merged to the merged area. 

55 [0221 ] When there is no adjacent initial area that can 
be merged to the merged area as shown in Fig. 19 (C), 
the area merging portion 84 extracts a portion com- 
posed of pixels that compose the foreground picture of 
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the n-th frame as an object corresponding to the object 
Obj of the (n-1)-th frame (this object may be referred to 
as corresponding object), assigns the same label as the 
label of the object Obj of the (n-1)-th frame, and outputs 
the object to the merged area process portion 85 and 
the separated area process portion 86. 
[0222] In other words, according to the embodiment, 
objects that correspond in individual frames are as- 
signed the same label. The receiving apparatus 2 rec- 
ognizes what object of a particular frame corresponds 
to what object of the adjacent frame corresponding to 
the label. 

[0223] The label assigned to the object Obj of the (n- 
1)-th frame is contained in the object information and 
stored in the object information memory 88. With refer- 
ence to the object information memory 88, the area 
merging portion 84 recognizes the label assigned to the 
object Obj of the (n-1)-th frame. 

[0224] In addition, after the area merging portion 84 
extracts objects of the n-th frame corresponding to all 
objects of the (n-1)-th frame in the above-described 
manner, the area merging portion 84 extracts an area 
of which each initial area of the n-th frame and another 
initial area adjacent thereto are merged as an object and 
assigns a new label (other than any label assigned to 
objects of the (n-1 )th frame) to the extracted object and 
outputs the extracted object to the merged area process 
portion 85 and the separated area process portion 86. 
[0225] The merged area process portion 85 and the 
separated area process portion 86 perform a merged 
area process and a separated area process, respective- 
ly. The merged area process is performed when objects 
are merged. The separated area process is performed 
when a merged object is separated. 
[0226] In other words, for example, considering three 
successive frames of the (n-1 )-th frame, the n-th frame, 
and the (n+1)-th frame, as shown in Fig. 20, in the (n- 
1)-th frame, objects A and B approach each other. In the 
n-th frame, the objects A and B overlap. In other words, 
in the n-th frame, the objects A and B may be merged 
as one object. In this case, the objects A and B that have 
been merged as one object continue to move in their 
directions. In the (n+1)-th frame, the merged object is 
separated into the two objects A and B. 
[0227] Inthiscase, In the area merging process of the 
area merging portion 84, the merged object of the n-th 
frame corresponds to both the objects A and B in the (n- 
1)-th frame. In contrast, the objects A and B separated 
in the (n+1)-th frame correspond to one merged object 
in the n-th frame. According to the embodiment, it is as- 
sumed that one object of a particular frame corresponds 
to one object of the immediately preceding frame. Thus, 
as was described above, it is not preferred to correlate 
two objects with one object. In contrast, it is not preferred 
to correlate one object with two objects. 
[0228] Thus, the merged area process portion 85 per- 
forms a merged area process for correlating one 
merged object of the n-th frame with one of the objects 



A and B of the (n-1 )-th frame. The separated area proc- 
ess portion 86 performs a separated area process for 
correlating one of two objects A and B of the (n+1)-th 
frame with one merged object of the n-th frame. 

5 [0229] In other words, the merged area process por- 
tion 85 assigns the same label as one of the objects A 
and B of the (n-1)-th frame to one merged object of the 
n-th frame. On the other hand, the separated area proc- 
ess portion 86 assigns the same label as the label of the 

10 merged object of the n-th frame to one of the two sepa- 
rated objects A and B of the (n+1 )-th frame and assigns 
a new label to the other separated object of the (n+1)- 
th frame. 

[0230] The object extracted result as the result of the 

is merged area process performed by the merged area 
process portion 85 and the object extracted result as the 
result of the separated area process performed by the 
separated area process portion 86 are merged by the 
area merging portion 84 and supplied to a new area 

20 process portion 87. 

[0231] When the object extracted results of the 
merged area process portion 85 and the separated area 
process portion 86 contain a new object, the new area 
process portion 87 performs a new area process for the 

25 new object. 

[0232] There are three types of objects that are as- 
signed new labels in the object extracted results of the 
merged area process portion 85 and the separated area 
process portion 86. The first type is an object of which 

30 since the motion of an object is fast and the object does 
not spatially overlap in the considered frame and the im- 
mediately preceding frame, the area merging portion 84 
does not extract the object of the considered frame as 
the corresponding object of the immediately preceding 

35 frame. The second type is an object of which since the 
corresponding object of the immediately preceding 
frame is merged with another object, the separated area 
process portion 86 cannot correlate the object of the 
considered frame with the corresponding frame of the 

40 immediately preceding frame. The th ird type is an object 
of which since a new object takes place in the consid- 
ered frame, a new label is assigned thereto. 
[0233] Among those three types of objects, a new la- 
bel should be assigned to an object of the third type. 

^5 Thus, for objects of the first and second types, the new 
area process portion 87 detects a corresponding object 
from the immediately preceding frame and reassigns 
the same label as the object of the considered frame to 
the corresponding object of the immediately preceding 

50 frame. 

[0234] In reality, the new area process portion 87 ref- 
erences the object information memory 88, recognizes 
objects of several past frames of the considered frame, 
and obtains the distance between each of the recog- 
55 nized objects and the considered object of the consid- 
ered frame, the considered object being assigned a new 
label. As the distance between objects, the distance be- 
tween feature amounts of the objects can be used. Ex- 
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am pies of a feature amount of an object are an area of 
the object, a histogram in the tangent direction of the 
contour line composed of each pixel of the object (for 
example, a histogram in each of eight directions of up, 
down, left, right, upper left, lower left, upper right, and 
lower right), and a moving vector of the object. 
[0235] The new area process portion 87 obtains the 
minimum value of the distances between the recognized 
objects and the considered object. When the minimum 
value is smaller than (or equal to) a predetermined 
threshold value, the new area process portion 87 treats 
an object with the minimum distance to the considered 
object as an object corresponding to the considered ob- 
ject, assigns the same label as the label of the obtained 
object to the considered object, and outputs the consid- 
ered object. When the minimum value of the distances 
between the recognized objects and the considered ob- 
ject is not smaller than the predetermined threshold val- 
jue (namely, there is no object whose distance to the con- 
^ sidered object is small in the past frames), the new area 
• process portion 87 treats the considered object as an 
object that newly takes place In the considered frame, 
assigns a new label to the considered object, and out- 
puts the considered object. 

[0236] An output of the new area process portion 87 
is supplied to the additional information calculating por- 
tion 1 5 (see Fig. 3) and the transmitting process portion 
16 (see Fig. 3). In addition, the output of the new area 
process portion 87 is supplied to the object information 
memory 88. The object information memory 88 tempo- 
rarily stores an. object (the position and size (contour) of 
the object, pixel values of pixels that compose the ob- 
ject, and so forth) and a label assigned thereto as object 
information. 

[0237] Next, with reference to a flow chart shown in 
Fig. 21 , the object extracting process for extracting an 
object from a picture will be described. The object ex- 
tracting process is performed by the object extracting 
^portion 14 shown in Fig. 17. 
}f [0238] A picture that is output by the picture inputting 
portion 11 and a background picture that is output by the 
background picture extracting portion 1 3 are supplied to 
the subtracting portion 81 . At step S41 , the subtracting 
portion 81 subtracts the background picture that is out- 
put from the background picture extracting portion 13 
from the picture that is output from the picture inputting 
portion 11 and obtains a foreground picture as an object. 
The foreground picture obtained by the subtracting por- 
tion 81 is supplied to the frame memory 82 and the initial 
area dividing portion 83. The frame memory 82 stores 
the foreground picture that is output from the subtracting 
portion 81 . 

[0239] On the other hand, at step S42, the initial area 
dividing portion 83 references a foreground picture of 
the immediately preceding frame of the considered 
frame, performs the initial area dividing process de- 
scribed with reference to Fig. 18, obtains initial areas, 
and supplies the obtained initial areas to the area merg- 



ing portion 84. At step S43, the area merging portion 84 
references object information of the immediately pre- 
ceding frame stored in the object information memory 
88, performs the area merging process as described 

5 with reference to Fig. 1 9 for the initial areas that are out- 
put from the initial area dividing portion 83, and extracts 
an object of the considered frame. 
[0240] The object extracted by the area merging por- 
tion 84 is supplied to the merged area process portion 

10 85 and the separated area process portion 86. At step 
S44 : the merged area process portion 85 or the sepa- 
rated area process portion 86 performs the merged area 
process or the separated area process as described 
with reference to Fig. 20 and outputs the processed re- 

15 suit to the new area process portion 87. 

[0241] At step S45, the new area process portion 87 
performs the above-described new area process for the 
outputs of the merged area process portion 85 and the 
separated area process portion 86. Thus, the new area 

20 process portion 87 outputs the final object extracted re- 
sult of the considered frame. The extracted result of the 
object is supplied to the additional information calculat- 
ing portion 15 (see Fig. 3) and the transmitting process 
portion 16 (see Fig. 3). In addition, the extracted result 

25 of the object is supplied to the object information mem- 
ory 88 and stored therein. 

[0242] Thereafter, the flow returns to step S20. At the 
step, the next frame is designated as a new considered 
frame. Thereafter, the object extracting portion 14 re- 
30 peats the similar process. 

[0243] Next, with reference to a flow chart shown in 
Fig. 22, the area merging process performed by the area 
merging portion 84 at step S43 of Fig. 21 will be de- 
scribed in detail. 

35 [0244] In the area margining process, first of all, at 
step S51 , the area merging portion 84 references object 
information about an object contained in the immediate- 
ly preceding frame (preceding frame) of the considered 
frame and treats the object of the preceding frame as a 

40 considered object. At step S51, using the considered 
object, as shown in Fig. 1 9 (B), the area merging portion 
84 merges initial areas that are output from the initial 
area dividing portion 83 and forms a merged area. 
[0245] Thereafter, the flow advances to step S52. At 

45 step S52, the area merging portion 84 searches an initial 
area adjacent to the merged area (this initial area may 
be referred to as adjacent initial area). Thereafter, the 
flow advances to step S53. At step S53, the area merg- 
ing portion 84 calculates the distance between the 

so merged area and the considered initial area. Thereafter, 
. the flow advances to step S54. At step S54, the area 
merging portion 84 determines whether or not the dis- 
tance between the areas is smaller than a predeter- 
mined threshold value. 

55 [0246] When the determined result at step S54 repre^ 
sents that the distance between the merged area and 
the considered initial area is smaller than the predeter- 
mined threshold value, the flow advances to step S55. 
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At step S55, the area merging portion 84 merges the 
considered initial area to the merged area and thereby, 
forms a new merged area. Thereafter, the flow advanc- 
es to step S56. 

[0247] In contrast, when the determined result at step 5 
S54 represents that the distance between the merged 
area and the considered initial area is not smaller than 
the predetermined threshold value, the flow advances 
to step S56, skipping step S55. In other words, the area 
merging portion 84 does not merge the considered initial 
area to the merged area. At step S56, the area merging 
portion 84 determines whether or not all initial areas ad- 
jacent to the merged area have been searched. When 
the determined result at step S56 represents that all the 
initial areas adjacent to the merged area have not been 
searched, the flow returns to step S52. At step S52, the 
area merging portion 84 searches an adjacent initial ar- 
ea that has not been searched. Thereafter, the area 
merging portion 84 repeats the similar process. 
[0248] When the determined result at step S56 repre- 
sents that all initial areas adjacent to the merged area 
have been searched, the flow advances to step S57. At 
step S57, the area merging portion 84 determines 
whether or not each of all objects contained in the pre- 
ceding frame has been designated as the considered 
object. When the determined result at step S57 repre- 
sents that each of all objects contained in the preceding 
frame has not been designated as the considered ob- 
ject, the flow returns to step S51 . At step S51 , the area 
merging portion 84 designates one of objects that are 
contained in the preceding frame as the considered ob- 
ject and repeats the similar process for the newly con- 
sidered object. 

[0249] In contrast, when the determined result at step 
S57 represents that each of all the objects contained in 
the preceding frame has been designated as the con- 
sidered object, the flow returns to the called process. 
[0250] Next, with reference to a flow chart shown in 
Fig. 23, the merged area process performed by the 
merged area process portion 85 at step S44 shown in 
Fig. 21. will be described in detail. 
[0251] First of all, at step S61 , the merged area proc- 
ess portion 85 designates a frame that is processed as 
a considered frame, references the object information 
memory 88, recogni2esthe number of objects contained 
in the immediately preceding frame (preceding frame) 
(namely, the number of objects of the preceding frame 
of the considered frame) in the condition that the objects 
of the preceding frame spatially overlap with the consid- 
ered object of the considered frame, and sets the 
number of objects to a variable N. 
[0252] Thereafter, the flow advances to step S62. At 
step S62, the merged area process portion 85 deter- 
mines whether or not the variable N is 2 or larger. When 
the variable N is smaller than 2 (namely, no object that 
spatially overlaps with the considered object is con- 
tained in the preceding frame or the number of the ob- 
jects is one), the flow advances to step S65, skipping 



steps S63 and S64. 

[0253] In contrast, when the determined result at step 
S62 represents that the variable N is 2 or larger (namely, 
two or more objects that spatially overlap with the con- 
sidered object are contained in the preceding frame), 
the flow advances to step S63. At step S63, the merged 
area process portion 85 calculates the distance be- 
tween the considered object and each of two or more 
objects of the preceding frame that spatially overlap with 
the considered object. Thereafter, the flow advances to 
step S64. 

[0254] At step S64, the merged area process portion 
85 selects an object with the minimum distance to the 
considered object from the objects obtained at step S63 
and assigns the same label as the label of the selected 
object to the considered object! 

[0255] Thereafter, the flow advances to step S65. At 
step S65, the merged area process portion 85 deter- 
mines whether or not each of all objects contained in the 
considered frame has been designated as the consid- 
ered object. When the determined result at step S65 
represents that each of all objects contained in the con- 
sidered frame has not been designated as the consid- 
ered object, the merged area process portion 85 desig- 
nates one of those objects as the considered object. 
Thereafter, the flow returns to step S61 . At step S61 , the 
merged area process portion 85 repeats the similar 
process. 

[0256] On the other hand, when the determined result 
at step S65 represents that each of all the objects con- 
tained in the considered frame has been designated as 
the considered object, the flow returns to the called proc- 
ess. 

[0257] Next, with reference to a flow chart shown in 
Fig. 24 : the separated area process performed by the 
separated area process portion 86 at step S44 shown 
in Fig. 21 will be described in detail. 
[0258] The separated area process portion 86 refer- 
ences the object information memory 88 and designates 
one of objects contained in the immediately preceding 
frame (preceding frame) of the considered frame as a 
considered object. In addition, at step S71 , the separat- 
ed area process portion 86 recognizes the number of 
objects of the considered frame corresponding to the 
considered object (the recognized objects are referred 
to as corresponding objects) and sets the number of ob- 
jects to the variable N. 

[0259] Thereafter, the flow advances to step S72. At 
step S72, the separated area process portion 86 deter- 
mines whether or not the variable N is 2 or larger. 
[0260] When the determined result at step S72 repre- 
sents that the variable N is not larger than 2 (namely, no 
object that spatially overlaps with the considered object 
is contained in the considered frame or the number of 
objects is one), the flow advances to step S76, skipping 
steps S73 to S75. 

[0261] In contrast, when the determined result at step 
S72 represents t h at th e variable N is 2 or larger (namely, 



15 



20 



25 



30 



35 



40 



45 



50 



20 




EP 1 120 968 A1 40 



39 

two or more objects that spatially overlap with the con- 
sidered object are contained in the considered frame 
(objects corresponding to the considered object), the 
flow advances to step S73. At step S73, the separated 
area process portion 86 calculates the distance be- 
tween each of those objects and the considered object. 
Thereafter, the flow advances to step S74. 
[0262] At step S74, the separated area process por- 
tion 86 selects an object with the minimum distance to 
the considered object from those objects and assigns 
the same label as the label of the selected object to the 
considered object. 

[0263] Thereafter, the flow advances to step S75. At . 
step S75, the separated area process portion 86 as- 
signs a new label to one of the objects that are not se- 
lected at step S74 (namely, objects except for an object 
with the minimum distance to the considered object). 
Thereafter, the flow advances to step S76. 
£0264] At step S76, the separated area process por- 
S tion 86 determines whether or not each of all objects 
■ contained in the preceding frame has been designated 
as the considered object. When the determined result 
at step S76 represents that each of all the objects con- 
tained in the preceding frame has not been designated 
as the considered object, the separated area process 
portion 86 designates one of those objects as the con- 
sidered object. Thereafter, the flow returns to step S71 . 
At step S71 , the separated area process portion 86 re- 
peats the similar process. 

[0265] When the determined result at step S76 repre- 
sents that each of all the objects contained in the pre- 
ceding frame has been designated as the considered 
object, the flow returns to the called process. 
[0266] In the above-described case, when the user 
designates a considered point with the control informa- 
tion inputting portion 24, the transmitting apparatus 1 
controls the transmission of data so that the spatial res- 
olution of a picture in a priority range containing the con- 
jsidered point is improved at the sacrifice of the spatial 
J> resolution of the picture. Alternatively, the transmitting 
apparatus 1 may learn a preference of the user, detect 
an object or the like that the user tends to see with a 
high spatial resolution corresponding to the learnt result, 
and control the transmission of data so that the object 
is displayed with a high spatial resolution. 
[0267] Fig. 25 shows an example of the structure of 
the controlling portion 35 shown in Fig. 7 in the case that 
the transmitting apparatus 1 performs such a transmis- 
sion control. 

[0268] A priority range designating portion 91 re- 
ceives a control signal transmitted from the receiving ap- 
paratus 2, designates a priority range in the above-de- 
scribed manner, and supplies the designated priority 
range to a selection controlling portion 92 and a feature 
amount extracting portion 93. 

[0269] The selection controlling portion 92 controls 
the selection of data of a background picture, an object, 
and additional information of the MUX 32 (see Fig 7). 



In otherwords, when the selection controlling portion 92 
receives the priority range from the priority range des- 
ignating portion 91, the selection controlling portion 92 
controls the MUX 32 (see Fig. 7) to improve the spatial 

5 resolution of the picture in the priority range at the sac- 
rifice of the temporal resolution of the picture. In addi- 
tion, when the selection controlling portion 92 receives 
a label from an object detecting portion 95, the selection 
controlling portion 92 controls the MUX 32 (see Fig. 7) 

10 to improve the spatial resolution of an object with the 
label at the sacrifice of the temporal resolution of the 
picture. 

[0270] The data amount calculating portion 34 (see 
Fig. 7) supplies the data rate of the multiplexed data that 

15 j s output from the MUX 32 to the selection controlling 
portion 92. The selection controlling portion 92 controls 
the selection of data of the MUX 32 so that the data rate 
of the multiplexed data does not exceed the transmis- 
sion rate of the transmission path 3. 

20 [0271] A background picture, an object, and addition- 
al information that are output by the pre-process portion 
12 (see Fig. 3) and a priority range that is output by the 
priority range designating portion 91 are supplied to the 
feature amount extracting portion 93. The feature 

25 amount extracting portion 93 extracts a feature amount 
of a picture in the priority range that is output from the 
priority range designating portion 91 . In otherwords, the 
feature amount extracting portion 93 extracts a feature 
amount of an object contained in the priority range so 

30 that the feature amount reflects the tendency of the pic- 
ture that the user is considering. 
[0272] In reality, for example, as shown in Fig. 26, the 
feature amount of an object of a particular person ex- 
tracted by the feature amount extracting portion 93 rep- 

35 resents that an object is a person, the motion of the ob- 
ject is uniform, the position in the depth direction of the 
object (depth) is foreground, the position of the object 
on the display is center, the object is moving (the object 
is a portion that is moving), the area of the object con- 

40 tains eyes, a nose, and a mouth (the area of the object 
is composed of eyes, a nose, and a mouth), the pattern 
of the object is a stripe pattern (the object is a stripped 
portion), the color of the object is red (the object is a red 
portion). 

45 [0273] The feature amount extracting portion 93 ob- 
tains a vector (having elements of feature amounts) of 
an extracted object (the vector may be referred to as 
feature amount vector) and increments the frequency of 
the histogram of the obtained feature amount vector 

so stored in the histogram storing portion 94 by 1 . 

[0274] A histogram storing portion 94 stores the his- 
togram of the feature amount vector as the learnt result 
of the preference of the user. 

[0275] The object detecting portion 95 detects an ob- 
55 ject from those supplied from the pre-process portion 12 
(see Fig. 3) so that a feature amount vector with the 
maximum frequency of the histogram is obtained from 
the histogram storing portion 94. In other words, the ob- 
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ject detecting portion 95 obtains a feature amount vector 
for an object supplied from the pre-process portion 12 
(see Fig. 3) in the same manner as the feature amount 
extracting portion 93. In addition, the object detecting 
portion 95 references the histogram stored in the histo- 
gram storing portion 94 and determines whether or not 
a feature amount vector of an object supplied from the 
pre-process portion 12 (see Fig. 3) is contained in a pre- 
determined range of the feature amount vector space 
around the feature amount vector with the highest fre- 
quency. When the feature amount vector is contained in 
the predetermined range, the object detecting portion 
95 designates the object as an object that the user tends 
to consider and supplies the label of the object to the 
selection controlling portion 92. 
[0276] Next, with reference to a flow chart shown in 
Fig. 27, the controlling process of the MUX 32 (see Fig. 
7) will be described. The controlling process is per- 
formed by the controlling portion 35 shown in Fig. 25. 
[0277] First of all, at step S81 , the priority range des- 
ignating portion 91 determines whether or not a control 
signal has been transmitted from the receiving appara- 
tus 2. When the determined result at step S81 repre- 
sents that a control signal has been transmitted from the 
receiving apparatus 2, the flow advances to step S82. 
At step S82, the priority range designating portion 91 
designates a priority range in the above-described man- 
ner corresponding to the control signal and supplies the 
priority range to the selection controlling portion 92 and 
the feature amount extracting portion 93. 
[0278] At step S83,. the selection controlling portion 
92 controls the MUX 32 (see Fig. 7) to improve the spa- 
tial resolution of a picture (an object and a background 
picture) in the priority range supplied from the priority 
range designating portion 91 at the sacrifice of the tem- 
poral resolution of the picture. 

[0279] At step S84, the feature amount extracting por- 
tion 93 extracts the feature amount of the object in the 
priority range supplied from the priority range designat- 
ing portion 91 and obtains a feature amount vector hav- 
ing elements composed of each feature amount of the 
object. At step S85, the feature amount extracting por- 
tion 93 increments the frequency of the histogram of the 
feature amount vector stored in the histogram storing 
portion 94 by 1. Thereafter, the flow returns to step S81. 
[0280] In the loop from step S81 to step S85, the his- 
togram storing portion 94 forms a histogram of a feature 
vector of an object that the user tends to consider. In 
other words, the histogram storing portion 94 learns the 
preference of the user. 

[0281] In addition, the feature amount extracting por- 
tion 93 can quantized the obtained feature amount vec- 
tor and increment the frequency of code corresponding 
to the quantized result of the feature amount vector. In 
this case, the histogram storing portion 94 temporarily 
stores the histogram of the code. 

[0282] In contrast, when the determined result at step 
S81 represents that the control signal has not been 



transmitted from the receiving apparatus 2, the flow ad- 
vances to step S86. At step S86, the object detecting 
portion 95 obtains a feature amount vector of the object 
supplied from the pre-process portion 12 (see Fig. 3) in 
5 the same manner as the feature amount extracting por- 
tion 93. In addition, at step S87, the object detecting por- 
tion 95 references the histogram stored in the histogram 
storing portion 94 and determines whether or not a fea- 
ture amount vector of the object supplied from the pre- 
process portion 12 (see Fig. 3) is contained in a prede- 
termined range of the feature amount vector space 
around the feature amount vector with the highest fre- 
quency In other words, at step S87, the object detecting 
portion 95 determines whether or not the distance be- 
tween the feature amount vector with the highest fre- 
quency and the feature amount vector of the object sup- 
plied from the pre-process portion 12 is equal to or 
smaller than a predetermined value. 
[0283] As was described above, when the histogram 
storing portion 94 stores the histogram of the code as 
the vector quantized result, the object detecting portion 
95 quantizes the obtained feature amount vector. At 
step S87, the object detecting portion 95 determines 
whether or not the code as the vector quantized result 
matches the code of the highest frequency of the histo- 
gram stored in the histogram storing portion 94. 
[0284] When the determined result at step S87 repre- 
sents that the distance between the feature amount vec- 
tor with the highest frequency and the feature amount 
vector of the object supplied from the pre-process por- 
tion 12 is not smaller than the predetermined value 
(namely, the object supplied from the pre-process por- 
tion 12 is an object that the user does not tend to con- 
sider due to his or her tendency), the flow advances to 
step S88. At step S88, the selection controlling portion 
92 controls the MUX 32 (see Fig. 7) so that the receiving 
apparatus 2 displays a picture with a regular temporal 
resolution and a regular spatial resolution. Thereafter, 
the flow returns to step S81 . 

[0285] In contrast, when the determined result at step 
S87 represents that the distance between the feature 
amount vector with the maximum frequency and the fea- 
ture amount vector of the object supplied from the pre- 
process portion 12 is equal to or smaller than the pre- 
determined value (namely, the object supplied from the 
pre-process portion 12 is an object that the user tends 
to consider due to his or her tendency), the object de- 
tecting portion 95 outputs the label of the object supplied 
from the pre-process portion 1 2 to the selection control- 
ling portion 92. Thereafter, the flow advances to step 
S89. 

[0286] At step S89, the selection controlling portion 
92 controls the MUX 32 (see Fig. 7) to improve the spa- 
tial resolution of the object with the label supplied from 
the object detecting portion 95 atthe sacrifice of thetem- 
poral resolution. Thereafter, the flow returns to step S81 . 
[0287] Thus, in this case, the receiving apparatus 2 
displays ah object with a label that is output from the 
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object detecting portion 95 so that the spatial resolution 
becomes high at the sacrif ice of the temporal reso ut.on. 
Thereafter, the receiving apparatus 2 continues to de- 
play the object with the high spatial resolution. 
[02881 As a result, the receiving apparatus 2 automat- 
ically displays an object that the user tends to consider 
so that the spatial resolution of the object becomes h gh 
without necessity of the intervention of the user for the 
control information inputting portion 24. Thereafter the 
receiving apparatus 2 continues to display the object 
with the high spatial resolution (however, in this case 
as was described above, the temporal resolution of the 

picture deteriorates). „~*« r 
[02891 The histogram of the feature amount vector 
stored as the learnt result of the preference of the user 
in the histogram storing portion 94 can be reset penod- 
ically, non-periodically, or corresponding to a request 
from the user of the receiving apparatus 2. 
■ [02901 in the above-described case, the spatial reso- 
lution of an object that has a feature amount vector that 
matches orthat is similar to afeature amount vector with 
the highest frequency of the histogram is improved^ Al- 
ternatively, spatial resolutions of all objects having fea- 
ture amount vectors that match or that are similar to a 
feature amount vector whose frequency in the histo- 
gram exceeds a predetermined value may be improved 
[02911 The priority range is a predeteimined range of 
which a considered point is at the center of gravity (,n 
the above-described case, the priority range is a rectan- 
gular range). However, it can be said that an area of a 
picture in a particular range containing a considered 
point is a picture area that the user wants to see with 
interest (hereinafter, this area is referred to as interested 

rSl^n the other hand, a moving picture displayed 
by the receiving apparatus 2 has a moving picture area 
(hereinafterthis area may be referredto as moving area) 
and a still picture area (hereinafterthis area may be re- 
' ferred to as still area), 
t. [02931 To improve the spatial resolution of the inter 
ested object area, it is necessary to recognize (desig- 
nate) an interested object area that the user is consid- 
ering in both the moving area and the still area of the 
picture. When the interested object area that the user is 
considering can be designated, a picture area that the 
user is not considering <for example, a background pic- 
ture) can be obtained. 

(-02941 Even if an interestedobject area that the user 
is considering can be designated at a particular time, 
thereafter, it may be changed. Thus, when a p.ct ure area 
that the user is considering is changed to another picture 
area, it is necessary to recognize the new picture area 
as the interested object area. .. , . + 

[0295] In addition, there may be a plurality of interest- 
ed object areas that the user is considering. In this case 
it is necessary to separately recognizes those interested 

objectareas. „^„„h 
[02961 Thus, Fig. 28 shows an example of a second 



structure of the picture transmitting system shown in Fig . 
1 in the case that portable terminal units are used as the 
transmitting apparatus 1 and the receiving apparatus 2 
shown in Fig.M. In Fig. 28, similar portions to those in 
Fig 2 are denoted by similar reference numerals and 
their description is omitted. In otherwords, the structure 
of the picture transmitting system shown in Fig. 28 is 
basically the same as the structure of the picture trans- 
mitting system shown in Fig. 2 . 

[02971 In the embodiment shown in Fig: 2, all informa- 
tion necessary for controlling a spatial resolution and a 
temporal resolution such as the coordinates of a con- 
sidered point thatthe user is considering, the spatial res- 
olution, and the transmission rate is transmitted as con- 
trol information from the receiving apparatus 2 to the 
transmitting apparatus 1 . In contrast, according to the 
embodiment shown in Fig. 28, as will be described later 
information of a considered point of a picture displayed 
on the displaying portion 2-2 that is operated (clicked) 
by the user with the key portion 2-3 of the receiving ap- 
paratus 2 is transmitted as control information (herein- 
after, information at a considered point may be referred 
to as click data). 

[02981 When the transmitting apparatus 1 receives 
click data from the receiving apparatus 2, the transmit- 
ting apparatus 1 designates a picture area (interested 
object area) that the user is considering from a picture . 
displayed by the receiving apparatus 2 (the picture is 
photographed by the video camera portion 1-1 of the 
transmuting apparatus ^corresponding to thechckda a 
and controls the information amount of the picture data 
that is transmitted to the receiving apparatus 2 so that „ 
the spatial resolution and the temporal resolution of the 
designated picture area are changed while a predeter- 
mined condition is satisfied . 

[0299] Next, Fig. 29 shows an example of the struc- 
ture of the transmitting apparatus 1 shown in Fig. 28 In 
Fig. 29. similar portions to those in Fig. 3 are denoted 
by similar reference numerals and their descnpt.on is 
omitted. In the transmitting apparatus 1 shown in Fig. 
29 a background picture extracting portion 1 01 3, an ob- 
ject extracting portion 1014, and a transmitting process 
portion 1016 are disposed instead of the background 
picture extracting portion 13, the object extracting por- 
tion 14,and the transmitting process portion 16, respec- 
tively in addition, click data transmitted from the receiv- 
ing apparatus 2 is supplied to not only the transmitting 
process portion 1 016 corresponding to the transmitting 
process portion 1 6, but the pre-process portion 12. Ex- 
cept for those points, the structure of the transmitting 
apparatus 1 shown in Fig. 29 is basically the same as 
the structure of the transmitting apparatus 1 shown in 
Fiq 3 However, in the transmitting apparatus 1 shown 
in Fig. 29, an output of the background picture extracting 
portion 1 01 3 is not supplied to the object extracting por- 
tion 1014 In contrast, in the transmitting apparatus 1 
shown in Fig. 3, an output of the background picture ex- 
tracting portion 13 is supplied to the object extracting 
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portion 14. In addition, in the transmitting apparatus 1 
shown in Fig. 29, an output of the object extracting por- . 
tion 1 01 4 is supplied to the background picture extract- 
ing portion 1 013. In contrast, in the transmitting appara- 
tus 1 shown in Fig. 3, an output of the object extracting 
portion 1 4 is not supplied to the background picture ex- 
tracting portion 13. 

[0300] Click data is supplied to the pre-process por- 
tion 12. In the pre-process portion 12, the click data is 
supplied to the object extracting portion 1014. The ob- 
ject extracting portion 1014 extracts (designates) a pic- 
ture area (interested object area) that the user of the 
receiving apparatus 2 is considering from a picture pho- 
tographed by the picture inputting portion 11 and sup- 
plies picture data corresponding to the extracted (des- 
ignated) interested object area to the transmitting proc- 
ess portion 1 01 6. When there are a plurality of interest- 
ed object areas that the user of the receiving apparatus 
2 is considering in the picture photographed by the pic- 
ture inputting portion 11, the object extracting portion 
1014 supplies picture data of the plurality of interested 
object areas to the transmitting process portion 101 6. In 
addition, the picture data of an interested object area 
extracted by the object extracting portion 1014 is also 
supplied to the additional information calculating portion 
15. 

[0301 ] As an example of an interested object area that 
the user is considering is an object such as a substance. 
Next, the case that an object (hereinafter, referred to as 
object picture) is extracted as an example of an inter- 
ested object area by the object extracting portion 1014 
will be described. It should be noted that an interested 
object area is not limited to an object. Instead, an inter- 
ested object area may be a picture area other than an 
object, a picture area in an object, or a background pic- 
ture portion (that will be described later). However, ac- 
cording to the embodiment, as an example, the case 
that an interested object area is an object will be de- 
scribed. The object extracting process (interested object 
area designating process) performed by the object ex- 
tracting portion 1014 will be described later. 
[0302] The background picture extracting portion 
1013 extracts a signal (hereinafter, referred to as back- 
ground picture data) corresponding to abackground pic- 
ture portion (that is a picture area other than an inter- 
ested object area; hereinafter, the background picture 
portion is referred to as background picture) of the pic- 
ture from picture data supplied from the picture inputting 
portion 11 corresponding to the object extracted result 
of the object extracting portion 1014 and supplies the 
extracted background picture data to the transmitting 
process portion 1016 and the additional information cal- 
culating portion 15. In this example, a plane picture area 
whose activity is low (namely, that does not have a sig- 
nificance as a picture) is treated as a background pic- 
ture. Of course, as well as a picture that does not have 
a significance, a background picture may be an object 
in which the user is not interested. In this example, for 



simplicity, the above-described plane picture area will 
be described as a background picture. 
[0303] The additional information calculating portion 
1 5 detects a background picture moving vector that rep- 

s resents the motion of a background picture correspond- 
ing to background picture data supplied from the back- 
ground picture extracting portion 1013 (the motion of the 
background picture corresponds to the motion in the 
photographing direction of the picture inputting portion 

10 11). in addition, the additional information calculating 
portion 15 detects an object moving vector that repre- 
sents the motion of an object corresponding to picture 
data of an object picture (hereinafter, this picture data 
is referred to as object picture data) supplied from the 

is object extracting portion 1014. The additional informa- 
tion calculating portion 15 supplies the detected moving 
vector as a part of the additional Information to the trans- 
mitting process portion 1016. Moreover, the additional 
information calculating portion 15 supplies information 

20 about the object such as the position and contour of the 
object of the picture (frame) photographed by the picture 
inputting portion 11 corresponding to the object picture 
data supplied from the object extracting portion 1 01 4 as 
additional information to the transmitting process por- 

25 tion 1016. In other words, when the object extracting 
portion 1014 extracts an object picture from picture data, 
the object extracting portion 1014 also extracts informa- 
tion about the object such as the position and contour 
of the object and supplies the information to the addi- 

30 tional information calculating portion 15. The additional 
information calculating portion 15 outputs information 
about the object as additional information. 
[0304] The transmitting process portion 1016 en- 
codes the object picture data supplied from the object 

35 extracting portion 1014, the background picture data 
supplied from the background picture extracting portion 
1013, and the additional information supplied from the 
additional information calculating portion 15 corre- 
sponding to the click data supplied from the receiving 

40 apparatus 2 so that while the spatial resolution of the 
object picture of the picture displayed by the receiving 
apparatus 2 is improved, the condition of the data rate 
of the transmission path 3 is satisfied. Thereafter, the 
transmitting process portion 1016 multiplexes the en- 

45 coded object picture data, the encoded background pic- 
ture data, and the encoded additional information and 
transmits the multiplexed data, the frame rate informa- 
tion, and so forth to the receiving apparatus 2 through 
the trahsmission path 3. 

so [0305] Next, with reference to a flow chart shown in 
Fig. 30, the process performed by the transmitting ap- 
paratus 1 shown in Fig. 29 will be described. 
[0306] At step S91 , the transmitting apparatus 1 in- 
puts picture data obtained by the picture inputting por- 

55 tion 11 to the pre-process portion 12. 

[0307] Thereafter, at step S92, the transmitting appa- 
ratus 1 receives click data transmitted from the receiving 
apparatus 2 and inputs the click data to the pre-process 
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portion 12. 

[0308] At step S93, the pre-process portion 12 that 
has received the picture data and the click data performs 
a pre-process for extracting a background picture, an 
object, and additional information and supplies the 
background picture data, object picture data, and addi- 
tional information obtained in the pre-process to the 
transmitting process portion 101 6. 
. [0309] At step S94, the transmitting process portion 
1016 calculates the data amounts of the object picture 
data, background picture data, and additional informa- 
tion so that the condition of the data rate of the trans- 
mission path 3 is satisfied. Thereafter, the transmitting 
process portion 1016 encodes the object picture data, 
background picture data, and additional information cor- 
responding to the data amounts and then multiplexes 
them. Thereafter, along with the frame rate information 
and so forth, the transmitting process portion 1016 
{transmits the multiplexed data to the receiving appara- 
*y tus 2 through the transmission path 3. 

[031 0] Thereafter, the flow returns to step S1 . At step 
S1, the transmitting apparatus 1 repeats the similar 
process. 

[0311] Next, Fig. 31 shows an example of the struc- 
ture of the receiving apparatus 2 shown in Fig. 28. In 
Fig. 31 , similar portions to those in Fig. 5 are denoted 
by similar reference numerals and their description is 
omitted. In other words, in the receiving apparatus 2 
shown in Fig. 31 , a combining process portion 1 022 is 
disposed instead of the combining process portion 22. 
in addition, a click data inputting portion 1 024 and a click 
data transmitting portion 1025 are disposed instead of 
the control information inputting portion 24 and the con- 
trol information transmitting portion 25, respectively. Ex- 
cept for those points, the structure of the receiving ap- 
paratus 2 shown in Fig. 28 is basically the same as the 
structure of the receiving apparatus 2 shown in Fig. 5. 
[0312] Multiplexed data transmitted from the transmit- 
ting apparatus 1 through the transmission path 3 is re- 
J : ceived by the receiving process portion 21 . The receiv- 
ing process portion 21 separates the received multi- 
plexed data into encoded background picture data, en- 
coded object picture data, and encoded additional infor- 
mation data and decodes the separated data. Thereaf- 
ter, the receiving process portion 21 supplies the decod- 
ed background picture data, object picture data, and ad- 
ditional information to the combining process portion 
1022. 

[0313] The combining process portion 1022 com- 
bines the decoded background picture data, object pic- 
ture data, and additional information and supplies the 
combined picture signal to the picture outputting portion 
23. In addition, the combining process portion 1 022 con- 
trols the spatial resolution and temporal resolution of the 
picture that is combined corresponding to click data sup- 
plied from the click data inputting portion 1 024. 
[031 4] The click data inputting portion 1 024 generates 
click data that represents the click position (coordinate 



position) and the click time corresponding to the opera- 
tion of the key portion 2-3 by the user. The key portion 
2-3 functions as a pointing device for designating the 
coordinate position of a picture displayed on the picture 

5 outputting portion 23 corresponding to the displaying 
portion 2-2 of the receiving apparatus 2 (see Fig. 28). In 
other words, when the user clicks the key portion 2-3 for 
a desired picture portion (interested object area) of a 
picture displayed on the picture outputting portion 23, 

10 the click data inputting portion 1 024 generates click data 
that represents the coordinate information of the click 
position and the click time. The click data generated by 
the click data inputting portion 1024 is sent to the com- 
bining process portion 1 022 and the click data transmit- 

15 ting portion 1025. 

[0315] When the click data transmitting portion 1025 
receives the click data from the click data inputting por- 
tion 1 024, the click data transmitting portion 1 025 trans- 
mits the click data to the transmitting apparatus 1 

20 through the transmission path 3. 

[0316] Next, with reference to a flow chart shown in 
Fig. 32. the process performed by the receiving appa- 
ratus 2 shown in Fig. 31 will be described in brief. 
[0317] First of all, at step S1 01 , the receiving appara- 

25 tus 2 receives multiplexed data transmitted from the 
transmitting apparatus 1 through the transmission path 
3. 

[0318] At step S1 02, the receiving process portion 21 
separates the multiplexed data into encoded back- 

30 ground picture data, encoded object picture data, and 
encoded additional information data and decodes the 
separated encoded data. The decoded background pic- 
ture data, object picture data, and additional information 
are sent to the combining process portion 1022. 

35 [0319] At step S103, in the receiving apparatus 2, the 
click data inputting portion 1 024 obtains click data cor- 
responding to a click operation of the key portion 2-3 by 
the user and supplies the click data to the combining 
process portion 1 022 and the click data transmitting por- 

40 tion 1025. Thus, the click data is transmitted from the 
click data transmitting portion 1025 to the transmitting 
apparatus 1. 

[0320] At step S104, the combining process portion 
1022 combines a picture and controls the spatial reso- 

^5 lution and the temporal resolution of the combined pic- 
ture corresponding to the background picture data, ob- 
ject picture data, and additional information supplied 
from the receiving process portion 21 and the click data 
supplied from the click data inputting portion 1 024. The 

so transmitting apparatus 1 can place the click data trans- 
mitted from the receiving apparatus 2 in header infor- 
mation of the multiplexed data and transmit the resultant 
header information to the receiving apparatus 2. In this 
case, the combining process portion 1 022 of the receiv- 

55 ing apparatus 2 can obtain the click data from the header 
information. Thus, it is not necessary to supply the click 
data from the click data inputting portion 1024 to the 
combining process portion 1022. 
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[0321] At step S1 05, the picture outputting portion 23 
displays the picture combined by the combining process 
portion 1 022 on a liquid crystal display or the like of the 
picture outputting portion 23. 

[0322] Thereafter, the flow returns to step S101. At 
step S1 01 , the receiving apparatus 2 repeats the similar 
process. 

[0323] Next, Fig. 33 shows a real example of the 
structure of the transmitting process portion 1 01 6 of the 
transmitting apparatus 1 shown in Fig. 29. In Fig. 33, 
similar portions to those in Fig. 7 are denoted by similar 
reference numerals and their description is omitted. In 
other words, the structure of the transmitting process 
portion 1016 shown in Fig. 33 is basically the same as 
the structure of the transmitting process portion 1016 
shown in Fig. 7 except that the transmitting process por- 
tion 1016 shown in Fig. 33 supplies click data (as part 
of control information) rather than the whole control in- 
formation to the controlling portion 35. 
[0324] In Fig. 33; background picture data, object pic- 
ture data, and additional information are supplied from 
the pre-process portion 1 2 shown in Fig. 29 to the trans- 
mitting process portion 1016. The background picture 
data, object picture data, and additional information are 
input to the encoding portion 31 and the controlling por- 
tion 35. The encoding portion 31 hierarchically encodes 
the supplied background picture data, object picture da- 
ta, and additional information in the above-described 
manner and supplies the obtained encoded data to the 
MUX 32. The MUX 32 selects the encoded background 
picture encoded data, encoded object picture encoded, 
and encoded additional information data under the con- 
trol of the controlling portion 35 and supplies the select- 
ed data as multiplexed data to the transmitting portion 
33. The transmitting portion 33 modulates the multi- 
plexed data supplied form the MUX 32 corresponding 
to the transmission standard of the transmission path 3 
as a downstream portion and transmits the modulated 
data to the receiving apparatus 2 through the transmis- 
sion path 3. 

[0325] On the other hand, the controlling portion 35 
controls an output of the multiplexed data supplied from 
the MUX 32 so that the data rate supplied from the data 
amount calculating portion 34 does not exceed the 
transmission rate of the transmission path 3. In addition, 
the controlling portion 35 receives click data transmitted 
from the receiving apparatus 2 through the transmission 
path 3 and controls the MUX 32 to select and multiplex 
the encoded data corresponding to the click data. 
[0326] Next, with reference to a flow chart shown in 
Fig. 34, the transmitting process performed by the trans- 
mitting process portion 1016 shown in Fig. 33 will be 
described. 

[0327*5 First of all, at step S11 1 , the controlling portion 
35 of the transmitting process portion 1016 determines 
whether or not click data has been transmitted from the 
receiving apparatus 2. When the determined result at 
step S111 represents that the click data has not been 



transmitted from the receiving apparatus 2 (namely, the 
controlling portion 35 has not received click data), the 
flow advances to step S112. At step S112, as with the 
case at step S22 shown in Fig. 1 0, the controlling portion 
5 35 controls the MUX 32 to select encoded background 
picture data, encoded object data, and encoded addi- 
tional information data and multiplex the selected data 
so that the receiving apparatus 2 can display a picture 
with a regular temporal resolution. 
to [0328] Thereafter, the flow advances to step S1 13. At 
step S113, the transmitting process portion 1016 trans- 
mits the multiplexed data supplied from the MUX 32 
from the transmitting portion 33 through the transmis- 
sion path 3. Thereafter, the flow returns to step S11 1 . 
*5 [0329] When the determined result at step.SH 1 rep- 
resents that click data has been transmitted from the re- 
ceiving apparatus 2 (namely, the controlling portion 35 
has received click data), the flow advances to step S1 1 4. 
At step S114, the controlling portion 35 recognizes the 
coordinates (click position) and click time of a consid- 
ered point that the user has designated with the key por- 
tion 2-3 of the receiving apparatus 2 corresponding to 
the click data. 

[0330] Thereafter, at step S1 15, the controlling portion 
35 designates an interested object area that the user of 
the receiving apparatus 2 side has considered corre- 
sponding to the coordinates (click position) and the click 
time of the considered point in the manner that will be 
described later, designates the interested object area as 
a priority range for a picture whose spatial resolution is 
improved with priority, and detects a picture in the prior- 
ity range and additional information thereof. In this case, 
a picture in the priority range is an object picture. A pic- 
ture in the non-priority range is for example a picture 
35 such as a background picture in a non -interested object 
area. 

[0331] Thereafter, atstep S1 16, the controlling portion 
35 controls the MUX 32 to select encoded data of a pic- 
ture in the priority range (object picture), a picture in the 

40 non-priority range (background picture), and additional 
information and multiplex them. In other words, when 
the controlling portion 35 receives click data from the 
receiving apparatus 2, as in the case at step S26 shown 
in Fig. 10, the controlling portion 35 controls the MUX 

45 32 so that the spatial resolution of the picture In the pri- 
ority range is improved at the sacrifice of the temporal 
resolution. 

[0332] In addition, atstep S1 1 6, the controlling portion 
35 controls the MUX 32 to insert high resolution infor- 
50 mation as information of the position and size of the pri- 
ority range into the additional information selected as 
multiplexed data. Thereafter, the flow advances to step 
S113. 

[0333] At step S1 1 3, the transmitting portion 33 trans- 
55 mits the multiplexed data that is output by the MUX 32 
through the transmission path 3. Thereafter, the flow re- 
turns to step S1 11 . 

[0334] As was described above, in the transmitting 
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process shown in Fig. 34, the similar process to the 
process shown in Fig. 10 is performed. Thus, when the 
user of the receiving apparatus 2 continuously operates 
the click data inputting portion 1 024 (for example, he or 
she continuously designates the same considered 
point), for a picture in the priority range that contains a 
considered point (an interested object area and an ob- 
ject picture), data for improving the spatial resolution is 
transmitted with priority Thus, the spatial resolution of 
the picture in the priority range that contains the consid- 
ered point is gradually improved. As a result, the picture 
in the priority range is more clearly displayed. In other 
words, a picture (an interested object area and an object 
picture) that the user on the receiving apparatus 2 side 
is considering is more clearly displayed. 
[0335] As described above, the transmission of pic- 
ture data is controlled so that the spatial resolution and 
the temporal resolution of the picture in the priority range 
;(an interested object area and an object picture) are 
l changed in the range of the resolutions corresponding 
to the transmission rate of the transmission path 3. 
Thus, in a limited transmission rate, the spatial resolu- 
tion of an object picture corresponding to a considered 
point displayed by the receiving apparatus 2 can be 
more improved. In other words, since the spatial reso- 
lution of the object picture in the priority range is im- 
proved at the sacrifice of the temporal resolution of the 
picture, the object picture displayed on the receiving ap- 
paratus 2 can be more clearly displayed at the limited 
transmission rate (namely, the spatial resolution can be 
more improved). 

[0336] Next, Fig. 35 shows an example of the struc- 
ture of the combining process portion 1 022 shown in Fig. 
31. In Fig. 35, similar portions to those in Fig. 13 are 
denoted by similar reference numerals and their de- 
scription is omitted. In other words, the background pic- 
ture flag memory 74 is not disposed in the combining 
process portion 1022. A combining portion 1077 rather 
)than the combining portion 77 is disposed in the com- 
) bining process portion 1 022. In addition, click data rath- 
er than control information is supplied to the combining 
portion 1077. Except for those points, the structure of 
the combining process portion 1 022 shown in Fig. 35 is 
basically the same as the structure of the combining por- 
tion 22 shown in Fig. 13. 

[0337] Referring to Fig. 35, background picture data 
that is output from the receiving process portion 21 (see 
Fig. 31) is input to a background picture writing portion 

71 . Object picture data that is output from the receiving 
process portion 21 is input to an object writing portion 

72. Additional information that is output from the receiv- 
ing process portion 21 is input to the background picture, 
writing portion 71 , the object writing portion 72, and the 
combining portion 1077. 

[0338] The background picture writing portion 71 suc- 
cessively writes the supplied background picture data 
to a background picture memory 73. However, in the 
embodiment shown in Fig. 35, the background picture 



flag memory 74 of the embodiment shown in Fig. 13 is 
not disposed. Thus, in Fig. 35, when the background pic- 
ture writing portion 71 writes background picture data to 
the background picture memory 73, the background pic- 
5 ture writing portion 71 does not reference a background 
picture flag. 

[0339] The combining portion 1077 reads a back- 
ground picture of the frame that is displayed at present 
time (current frame) from the background picture data 

10 stored in the background picture memory 73 corre- 
sponding to a background picture moving vector con- 
tained in the additional information, combines an object 
picture stored in an object memory 75 and the back- 
ground picture corresponding to an object moving vec- 

15 tor contained in.the additional information, and supplies 
the combined picture of the current frame to a display 
memory 78. 

[0340] In addition, when the combining portion 1077 
receives click data from the click data inputting portion 

20 1024 shown in Fig. 31, the combining portion 1077 
reads object picture data that contains the coordinate 
position of a considered point contained in the click data 
from an object memory 75 and supplies the obtained 
object picture data to a sub window memory 79. 

25 [0341] Next, with reference to a flow chart shown in 
Fig. 36, the process (combining process) performed by 
the combining process portion 1022 shown in Fig. 35 
will be described. ' 

[0342] First of all, at step S1 21 , the object writing por- 
30 tion 72 writes object picture data supplied form the de- 
coding portion 53 shown in Fig. 35 in the above-de- 
scribed manner corresponding to an object flag stored 
in the object memory 75. 

[0343] Thereafter, the flow advances to step S122. 

35 The object writing portion 72 determines whether or not 
the additional information contains high resolution infor- 
mation. When the determined result at step S122 rep- 
resents that the additional information contains high res- 
olution information (namely, the user of the receiving ap- 

40 paratus 2 has operated the key portion 2-3, the click data 
• has been transmitted to the transmitting apparatus 1 , 
and then for a picture in the priority range, object picture 
data with a high spatial resolution has been transmitted 
from the transmitting apparatus 1 ), the flow advances to 

45 step S123. At step S1 23, the object writing portion 72 
sets a relevant object flag of the object flag memory 76 
to "I". 

[0344] In other words, when object picture data with 
a high spatial resolution for a picture in the priority range 

so has been transmitted from the transmitti ng apparatus 1 , 
at step S1 21 , the object picture data with the high spatial 
resolution is written to the object memory 75. Thus, at 
step S1 23, the object flag of the pixel that composes the 
object picture with the high spatial resolution is set to "I". 

55 [0345] Thereafter, the flow advances to step S1 24. At 
step S124, the combining portion 1 077 reads the object 
picture data in the priority range from the object memory 
75 and writes the obtained object picture data to the sub 
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window memory 79. 

[0346] In other words, when the determined result at 
step S122 represents that the additional information 
contains the high resolution information, as was de- 
scribed above, the use has operated the key portion 2-3, 
the click data has been transmitted to the transmitting 
apparatus 1, and then object picture data with a high 
spatial resolution for a picture in the priority range has 
been transmitted from the transmitting apparatus 1 . The 
click data transmitted to the transmitting apparatus 1 is 
also supplied to the combining portion 1077. When the 
combining portion 1 077 receives the click data, at step 
S1 24, the combining portion 1 077 recognizes the prior- 
ity range corresponding to tho coordinates and click time 
of the considered point contained in the click data, reads 
an object with a high spatial resolution in the priority 
range that has transmitted from the transmitting appa- 
ratus 1 from the object memory 75, and writes the ob- 
tained object to the sub window memory 79. In addition, 
as was described above, when the header information 
transmitted from the transmitting apparatus 1 contains 
click data, the combining portion 1 077 can recognize the 
priority range from the click data contained in the header 
information. 

[0347] Thereafter, the flow advances to step S1 25. At 
step S125, the combining portion 1077 reads back- 
ground picture data of the current frame from the back- 
ground picture memory 73 corresponding to a back- 
ground picture moving vector contained in the additional 
information. In addition, the combining portion 1077 
reads object picture data of the current frame that is dis- 
played from the object memory 75. Thereafter, the com- 
bining portion 1077 combines the background picture 
data of the current frame and the object picture data that 
has been read from the object memory 75 correspond- 
ing to an object moving vector contained in the addition- 
al information and writs the combined picture of the cur- 
rent frame to the display memory 78. In other words, the 
combining portion 1077 writes the background picture 
data to the display memory 78 and then overwrites the 
object picture data to the display memory 78. As a result, 
the combining portion 1 077 writes picture data of the 
current frame of which a background picture and an ob- 
ject picture have been combined to the display memory 
78. 

[0348] In the above-described manner, the picture da- 
La of the current frame written to the display memory 78 
and the object picture data written to the sub window 
memory 79 are supplied to the picture outputting portion 
23 shown in Fig. 31 and displayed thereon. 
[0349] In contrast, when the determined result at step 
S122 represents thatthe additional information does not 
contain high resolution information (namely, the user of 
the receiving apparatus 2 has not operated the key por- 
tion 2-3) . the flow advances to step S1 25, skipping steps 
S1 23 and S 124. As was described above, at step S 125, 
the combining portion 1077 reads background picture 
data of the current frame from the background picture 



memory 73, reads required object picture data from the 
object memory 75, and combines the background pic- 
ture of the current frame and the object picture that has 
been read from the object memory 75 corresponding to 

s the additional information. As a result, picture data of 
the current frame is formed and written to the display 
memory 78. Thereafter, the flow returns to step S121. 
At step S121, the combining portion 1077 repeats the 
similar process. 

w [0350] According to the above-described combining 
process, in the same manner as the case described with 
reference to Fig. 15, a picture with a high. spatial reso- 
lution that the user is considering as an object is dis- 
played. The combining process portion 1022 shown in 

*s Fig. 35 does not have the background picture flag mem- 
ory 74 shown in Fig. 13. Thus, the background picture 
writing portion 71 shown in Fig. 35 always writes sup- 
plied background picture data to the background picture 
memory 73. Thus, in the combining process shown in 

20 Rg. 36, the spatial resolution of the background picture 
is not improved unlike in the case described with refer- 
ence to Figs. 13 to 15. 

[0351] Next, a method for extracting an object picture 
(interested object area) corresponding to click datasup- 

25 plied from the receiving apparatus 2 will be described. 
This method is performed by the an object extracting 
portion 1 044 shown in Fig. 29. . 
[0352] Fig. 37 shows an example of the structure of 
the object extracting portion 1014 of the pre-process 

30 portion 12 shown in Fig. 29. 

[0353] In Fig. 37, picture data supplied from the pic- 
ture inputting portion 11 shown in Fig. 29 is stored to a 
picture memory 201 . The picture data is read from the 
picture memory 201 and supplied to a common terminal 

35 of a still area and moving area determining portion 203, 
an object picture extracting portion 21 3, and a selecting 
switch 207. The picture memory 201 stores picture data 
of at least several frames necessary for the still area and 
moving area determination of the still area and moving 

40 area determining portion 203 as a downstream portion 
of the picture memory 201 . 

[0354] In addition, click data transmitted from the re- 
ceiving apparatus 2 through the transmission path 3 is 
stored to a click data memory 202. The click data is read 

45 from the click data memory 202 and supplied to a com- 
mon terminal of a still area and moving area determining 
portion 204, a continuous click determining portion 204, 
and a selecting switch 206. The click data memory 202 
stores click data for a predetermined time period (for ex- 

50 ample, longer than 500 to 700 msec) necessary for the 
continuous click determination performed by the contin- 
uous click determining portion 204 as «a downstream 
portion of the click data memory 202. 
[0355] The still area and moving area determining 

55 portion 203 determines whether a picture area of a local 
small block (of for example 16x16 pixels) around the 
click position (coordinate value on a picture) represent- 
ed by the current click data sent from the receiving ap- 
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paratus 2 is a moving area or a still area, in other words, 
the still area and moving area determining portion 203 
obtains the difference between the picture area of the 
current frame and the picture area of a past frame that 
precedes the current frame by several frames (herein- 
after, this past frame is referred to as past frame) for 1 6 
x 16 pixels around the click position. When the differ- 
ence between the frames is equal to or smaller than a 
predetermined threshold value, the still area and moving 
area determining portion 203 determines that the picture 
area is a still area; In contrast, when the difference is 
more than the predetermined threshold value, the stili 
area and moving area determining portion 203 deter- 
mines that the picture area is a moving area. When a 
color picture is handled, the still area and moving area 
determining portion 203 obtains the difference between 
the picture area of the current frame and the picture area 
of the past frame for a picture of 16 x 16 pixels for each 
} of R, G, and B. When the average value of the absolute 

y values of the frame differences obtained for R, G, and 
B is equal to or smaller than a predetermined threshold 
value (for example, equal to or less than 10), the still 
area and moving area determining portion 203 deter- 
mines the picture area is a still area. In contrast, when 
the average value is larger than the predetermined 
threshold value, the still area and moving area deter- 
mining portion 203 determines that the picture area is a 
moving area. When the still area and moving area de- 
termining portion 203 determines that the picture area 
is a still area, the still area and moving area determining 
portion 203 determines that the current click data that is 
output from the click data memory 202 is a still click (a 
click in a still area). When the still area and moving area 
determining portion 203 determines that the picture area 
is a rnoving area, the stillarea and moving area deter- 
mining portion 203 determines that the current click data 
that is output from the click data memory 202 is a moving 
click (a click in a moving area). Thereafter the still area 
)and moving area determining portion 203 sends infor- 

) mation that represents a still click or a moving click as 
a still area and moving area determined result to a proc- 
ess determining portion 205. 

[0356] The continuous click determining portion 204 
determines whether or not the user of the receiving ap- 
paratus 2 has continuously performed the click opera- 
tion corresponding to the click time of the click data sent 
from Ihe receiving apparatus 2. In other words, the con- 
tinuous click determining portion 204 obtains the time 
difference between the click time of the current click data 
sent from the receiving apparatus 2 and the click time 
of the immediately preceding click data (namely, the 
click time interval) . When the time difference is equal to 
or smaller than a predetermined threshold value, the 
continuous click determining portion 204 determines 
that the user has not continuously performed the click 
operation. When the continuous click determining por- 
tion 204 determines that the user has continuously per- 
formed the click operation, the continuous click deter- 



mining portion 204 treats the current click data that is 
output form the click data memory 202 as a continuous 
click. In contrast, when the continuous click determining 
portion 204 determines that the user has not continu- 

5 ously perform the click operation (namely, the time dif- 
ference between the current click time and the immedi- 
ately preceding click time is equal to or larger than the 
predetermined threshold value), the continuous click 
determining portion 204 treats the current click data that 

10 is output from the click data memory 202 as a non-con- 
tinuous click. Thereafter, the continuous click determin- 
ing portion 204 sends information that represents a con- 
tinuous click or a non-corttinuous click as a continuous 
click determined result to the process determining por- 

15 tion205. 

[0357] The process determining portion 205 controls 
the selecting switch 206, the selecting switch 207, and 
a selecting switch 208 corresponding to the still area and 
moving area determined result of the still area and mov- 
20 }pg area determining portion 203 and the continuous 
click determined result of the continuous click determin- 
ing portion 204. 

[0358] In other words, for example, corresponding to 
the still area and moving area determined result and the 

25 continuous click determined result, when the current 
click data that is output from the click data memory 202 
is a still click and a continuous click, the process deter- 
mining portion 205 controls the selecting switch 206 so 
that the current click data that is output from the click 

30 data memory 202 is sent to a still object connecting proc- 
ess portion 211. In addition, the process determining 
portion 205 controls the selecting switch 207 so that the 
picture data that is output from the picture memory 201 
is sent to the still object connecting process portion 21 1 . 

35 Moreover, the process determining portion 205 controls 
the selecting switch 208 so that the immediately preced- 
ing click data that is output from an object extracted re- 
sult memory 21 4 (that will be described later), an object 
number (that corresponds to the above-described label 

40 and that categorizes (identifies) an object) assigned to 
the click data, and object picture data corresponding to 
the object number are sent to the still object connecting 
process portion 211. 

[0359] In addition, corresponding to the still area and 
45 moving area determined result and the continuous click 
determined result, when the current click data that is out- 
put from the click data memory 202 is a moving click 
and a continuous click, the process determining portion 
205 controls the selecting switch 206 so that the current 
50 click data that is output from the click data memory 202 
is sent to a moving object connecting process portion 
210. In addition, the process determining portion 205 
controls the selecting switch 207 so that picture data that 
is output from the picture memory 201 is sent to the mov- 
55 jng object connecting process portion 210. Moreover, 
the process determining portion 205 controls the select- 
ing switch 208 so that the immediately preceding click 
data that is output from the object extracted result mem- 
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ory 21 4 (that will be described later), the object number 
assigned to the click data, and the object picture data 
corresponding to the object number.are sent to the mov- 
ing object connecting process portion 210. 
[0360] In addition, corresponding to the still area and 
moving area determined result and the continuous click 
determined result, when the current click data that is out- 
put from the click data memory 202 is a still click and a 
continuous click (the time difference of the current click 
time and the immediately preceding click time is equal 
to or larger than the predetermined threshold value), the 
process determining portion 205 controls the selecting 
switch 206 so that the current click data that is output 
from the click data memory 202 is sent to an object 
number assigning portion 209. In addition, the process 
determining portion 205 controls' the selecting switch 
207 so that the picture datathat is output from the picture 
memory 201 is sent to the still object connecting process 
portion 21 1 . At that point, the selecting switch 208 con- 
trols the selecting switch 208 so that the immediately 
preceding click data that is output from the object ex- 
tracted result memory 214, the object number, and the 
object picture data are not sent to the still object con- 
necting process portion 211 (in this case, for example, 
the selecting switch 208 is opened). 
[0361] In addition, corresponding to the still area and 
moving area determined result and the continuous click 
determined result, when the current click data that is out- 
put from the click data memory 202 is a moving click 
and a non -contin uous click (the time difference between 
the current click time and the immediately preceding 
click time is equal to or larger than the predetermined 
threshold value), the process determining portion 205 
controls the selecting switch 206 so that the current click 
data that is output from the click data memory 202 is not 
sent to the object number assigning portion 209. In ad- 
dition, the process determining portion 205 controls the 
selecting switch 207 so that the picture data that'is out- 
put from the picture memory 201 is sent to the moving 
object connecting process portion 210: At that point, the 
process determining portion 205 controls the selecting 
switch 208 so that the immediately preceding click data 
that is output from the object extracted result memory 
214, the object number/and the object picture data are 
not sent to the moving object connecting process portion 
210 (for example, the selecting switch 208 is opened). 
[0362] The object number assigning portion 209 as- 
signs a new object number to click data as a non-con- 
tinuous click other than continuous clicks that are proc- 
essed as a connecting process by the still object con- 
necting process portion 21 1 and the moving object con- 
necting process portion 21 0 (thatwill be described later) 
and sends the object number and the click data to an 
object number memory 212. 

[0363] When the process determining portion 205 de- 
termines that the current click data that is output from 
the click data memory 202 is a moving click and a con- 
tinuous click, the moving object connecting process por- 



tion 21 0 determines whether the immediately preceding 
click data is a moving click and the feature of the picture 
in the vicinity of the current click position is contained in 
the feature of the area of the moving object picture with 
5 the object number assigned to the immediately preced- 
ing click data orthey are similarto each other. When the 
determined result is true, the moving object connecting 
process portion 21 0 determines that the current click is 
a click for the same object picture. Thus, the moving ob- 
10 ject connecting process portion 21 0 performs a moving 
object connecting process for assigning the same object 
number as the immediately preceding click data to the 
current click data and sends the object number and the 
click data to the object number memory 21 2. 
is [0364] When the determined result of the process de- 
termining portion 205 represents that the current click 
data that is output from the click data memory 202 is a 
still click and a continuous click, the still object connect- 
ing process portion 211 determines whether the imme- 
diately preceding click is a still click and the current click 
position is contained in the area of the still object picture 
with the object number assigned to the immediately pre- 
ceding click data or the current click position is close to 
the area. When the determined result of the still object 
connecting process portion 211 is true, the still object 
connecting process portion 211 determines that the cur- 
rent click is a click for the same object picture as the 
immediately preceding click. Thus, the still object con- 
necting process portion 211 performs a still object con- 
necting process for assigning the same object number 
as the immediately preceding click data to the current 
click data and sends the object number and the click 
data to the object number memory 212. 
[0365] The object number memory 212 stores click 
data for a plurality of past frames to which the object 
number assigning portion 209, the moving object con- 
necting process portion 21 0, and the still object connect- 
ing process portion 211 have assigned object numbers 
and sends the stored click data and object numbers to 
the object picture extracting portion 213. 
[0366] The object picture extracting portion 213 ex- 
tracts a still object picture, a moving object picture, a 
background picture, and so forth from the picture data 
supplied from the picture memory 201 corresponding to 
click data of the plurality of past frames that have been 
assigned object numbers and that have been supplied 
from the object number memory 212 and supplies the 
extracted result to the object extracted result memory 
214. 

[0367] In other words, the object picture extracting 
portion 213 obtains a dominant object number from a 
picture portion of click data with a high click data density 
as still clicks corresponding to click data of the plurality 
of past frames that have been assigned object numbers 
and that have been supplied from the object picture ex- 
tracting portion 213. The object picture extracting por- 
tion 213 forms the shape of an object corresponding to 
the distribution of the click data assigned with the dom- 
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inant object number and extracts a picture in the shape 
of the formed object as an object picture from the picture 
data. 

[0368] In addition , the object picture extracting portion 
213 performs a pattern matching operation for pictures 
of the frames in the vicinity of the click position assigned 
the same object number in click data determined as a 
moving click and performs a motion compensation for 
the pictures corresponding to the matching result. More- 
over, the object picture extracting portion 213 obtains a 
dominant object number of a picture area with a high 
click density from a picture area determined as a similar 
picture area (namely, a picture area that has been 
aligned by the motion compensation); The object picture 
extracting portion 213 forms the shape of an object cor- 
. responding to the distribution of the click data assigned 
the dominant object number and extracts a picture in the 
formed shape as an object picture from the picture data. 
{0369] In addition, the object picture extracting portion 
\ 213designatesapictureportion withalowstiilclick den- 
sity or a low moving click density as a background pic- 
ture. 

[0370] Along with click data, object number, and so 
forth, the object extracted result memory 21 4 stores ob- 
ject picture data extracted by the object picture extract- 
ing portion 213. When necessary, the object extracted 
result memory 214 supplies the object picture data to 
the background picture extracting portion 1013, the ad- 
ditional information calculating portion 15, and the trans- 
mitting process portion 1016 shown in Fig. 29. 
[0371] Next, with reference to a flow chart shown in 
Fig. 38, a process for extracting an object picture (inter- 
ested object area) that the user of the receiving appa- 
ratus 2 is considering from a picture that is being pho- 
tographed corresponding to click data transmitted from 
the receiving apparatus 2 will be described. The process 
is performed by the object extracting portion 1014 
shown in Fig 37. 

[0372] First of all, at step S131 , the picture memory 
) 201 stores picture data of a frame that is input from the 
.picture inputting portion 11 (frame picture data is input 
whenever transmitted). The picture memory 201 stores 
picture data of at least several frames necessary for the 
still area and moving area determining process per- 
formed at step S133. 

[0373] At step S131 / when click data is transmitted 
from the receiving apparatus 2 through the transmission 
path 3, the click data memory 202 stores the click data. 
The click data memory 202 stores click data for at least 
a predetermined time period (for example, longer than 
500 to 700 msec) necessary for the continuous click de- 
termining process performed at step S133 (that will be 
described later). 

[0374] Thereafter, the flow advances to step S1 32, At 
step S132, the object extracting portion 1014 deter- 
mines whether or not the click data memory 202 stores 
click data that has been transmitted from the receiving 
apparatus 2 and that has not been processed. When the 



determined result at step S1 32 represents that the click 
data memory 202 does not store click data that has not 
been processed, the flow returns to step S131 . At step 
S1 31 , the object extracting portion 101 4 waits for picture 

5 data and click data that are input. In contrast, when the 
determined result at step S1 32 represents that the click 
data memory 202 stores click data that has not been 
processed, the flow advances to step S133. At step 
S1 33, the object extracting portion 1 01 4 designates the 

10 oldest click data that has not been processed as the cur- 
rent click data. The still area and moving area determin- 
ing portion 203, the continuous click determining portion 
204, and the process determining portion 205 perform 
the still area and moving area determining process and 

is the continuous click determining process for the current 
click data. 

[0375] In other words, at step S133, the still area and 
moving area determining portion 203 performs the still 
area and moving area determining process for deter- 

20 mining whether a picture area as a local small block 
around a click position is a moving area or a still area 
using click position (picture coordinate value) informa- 
tion contained in the current click data transmitted from 
the receiving apparatus 2. 

25 [0376] Next, the still area and moving area determin- 
ing process at step S1 33 shown in Fig. 38 performed by 
the still area and moving area determining portion 203 
will be described in more reality. As shown in aflow chart 
shown in Fig. 39, at step S1 41 , the still area and moving 

so area determining portion 203 performs reads picture da- 
ta and click data for several frames from the picture 
memory 201 and the click data memory 202, respective- 
ly When a color picture is handled, as shown in Fig. 40 
(A), the still area and moving area determining portion 

35 203 reads picture data of several frames for R (Red), G 
(Green), and B (Blue). In addition, the still area and mov- 
ing area determining portion 203 reads picture data of 
several past frames containing a frame corresponding 
to the, current click data and click data corresponding to 

40 clicks performed for the several frames from the picture 
data memory 201 and the click data memory 202, re- 
spectively. 

[0377] Thereafter, at step S 142, the still area and 
moving area determining portion 203 calculates the dif- 

^ fere nee between a picture area of the current frame and 
a picture area of a past frame that precedes the current 
frame for several frame (this past frame is referred to as 
past frame) for a local small block composed of 1 6 x 1 6 
pixels in the horizontal direction and the vertical direc- 

50 tion around the click position of the current click data. 
When a color picture is handled, at step S142, as shown 
in Figs. 40 (B) and 40 (C), the still area and moving area 
determining portion 203 obtains the difference between 
the frames for a picture composed of 1 6 x 16 pixels for 

55 each of R, G, and B and obtains the average value of 
the absolute values of the differences for each of R, G, 
and B. 

[0378] Thereafter, at step S143, the still area and 
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moving area determining portion 203 determines wheth- 
er or not the difference between the frames calculated 
at step S1 42 is equal to or smaller than a predetermined 
threshold value. Thereafter, the flow advances to step 
S144. When the difference between the frames is equal 
to or smaller than the predetermined threshold value, 
the still area and moving area determining portion 203 
determines that a small block that contains the click po- 
sition of the current click data (this small block may be 
referred to as current block) is a still area. In contrast, 
when the difference between the frames is larger than 
the predetermined threshold value, the still area and 
moving area determining portion 203 determines that 
the current block is a moving area. In addition, when the 
determined result of the still area and moving area de- 
termining portion 203 represents that the current block 
is a still area, the still area and moving area determining 
portion 203 designates click data corresponding to the 
picture area of the current block as a still click. In con- 
trast, when still area and moving area determining por- 
tion 203 represents that the current block is a moving 
area, the still area and moving area determining portion 
203 designates click data corresponding to the picture 
area of the current block as a moving click. The still area 
and moving area determining portion. 203 outputs the 
determined result as a still area and moving area deter- 
mined result. 

[0379] In the case that a color picture is handled, at 
step S144, as shown in Fig. 40 (D), when the average 
of the absolute values of the differences between the 
frames for each block of 1 6 x 1 6 pixels for each of R, G, 
and B is equal to or smaller than a predetermined 
threshold value (for example, "10"), the still area and 
moving area determining portion 203 sets a predeter- 
mined flag to for example "0". In contrast, when the av- 
erage value is larger than the predetermined threshold 
value, the still area and moving area determining portion 
203 sets the predetermined flag to for example "I". At 
step S144, as shown in Fig. 40 (E), when all the flags 
for R, G, and B for the current block of 16 x 16 pixels 
are "0", the still area and moving area determining por- 
tion 203 determines that the current block is a still area 
and designates the click data corresponding to the pic- 
ture area of the current block as a still click. In contrast, 
when one of the flags is T, the still area and moving 
area determining portion 203 determines that the cur- 
rent block is a moving area and designates the click data 
corresponding to the picture area of the current block as 
a moving click. The still area and moving area determin- 
ing portion 203 outputs information of a still click or a 
moving click as a still area and moving area determined 
result. 

[0380] Returning to Fig. 38, at step S133, the contin- 
uous click determining portion 204 performs a continu- 
ous click determination for determining whether or not 
the click operation performed by the user of the receiv- 
ing apparatus 2 is a continuous click operation corre- 
sponding to click time contained in the click data trans- 



mitted from the receiving apparatus 2. 
[0381] Next, the process of the continuous click de- 
termining portion 204 performed at step S133 shown in 
Fig. 38 will be described. With reference to a flow chart 

s shown in Fig. 41, at step S151, the click data memory 
202 reads click data stored in the click data memory 202. 
[0382] Thereafter, at step S152, the continuous click 
determining portion 204 calculates the time difference 
(click interval) between the click time of the current click 

10 data transmitted from the receiving apparatus 2 and the 
click time of the immediately preceding click data (pre- 
ceding time). 

[0383] Next, at step S153, the continuous click deter- 
mining portion 204 determines whether or not the time 

if" difference is equal to or smaller than a predetermined 
threshold value. When the determined result at step 
SI 53 represents that the time difference is equal to or 
smaller than the predetermined threshold value, the 
continuous click determining portion 204 determines 

20 that the current click data is a continuous click. In con- 
trast, when the determined result at step S153 repre- 
sents that the time difference is larger than the prede- 
termined threshold value, the continuous click determin- 
ing portion 204 determines that the current click data is 

25 not a continuous click. Thereafter, the flow advances to 
step S154. At step S1 54, the continuous click determin- 
ing portion 204 outputs information that represents a 
continuous click or a non-continuous click as the con- 
tinuous click determined result. 

30 [0384] In other words, when the determined result at 
step S153 represents that the current click data is a con- 
tinuous click, the userof the receiving apparatus 2 tends 
to continuously perform the click operation for one pb-. 
ject picture. This is because when the user of the receiv- 
es jng apparatus 2 requests the transmitting apparatus 1 
to transmit object picture data (data in the interested ob- 
ject area) with a high spatial resolution, he or she tends 
to continuously click an object picture portion (interested 
object area) that the user wants to improve the spatial 

^0 resolution thereof. Thus, when the determined result of 
the continuous click determining portion 204 represents 
that the current click operation is a continuous click op- 
eration, the continuous click determining portion 204 
designates the current click data as a continuous click. 

45 in contrast, when the determined result of the continu- 
ous click determining portion 204 represents that the 
current click operation is not a continuous click opera- 
tion (namely, the time difference between the click time 
of the current click and the click time of the immediately 

so preceding click is equal to or larger than the predeter- 
mined threshold value), the continuous click determin- 
ing portion 204 designates the current click data as a 
non-continuous click. The continuous click determining 
portion 204 outputs the continuous click determined re- 

55 suit. 

[0385] Returning to Fig. 38, when the determined re- 
sults of the still area and moving area determining por- 
tion 203 and the continuous click determining portion 
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204 at step S1 33 represent that the current click data is 
a still click and a continuous click, the process determin- 
ing portion 205 controls the selecting switch 206, the.se- 
lectlng switch 207, and the selecting switch 208 in the 
above-described manner. Thus, at step S135, the still 
object connecting process portion 211 performs a still 
object connecting process. When the determined re- 
sults represent that the current click data is a moving 
click and a continuous click, the process determining 
portion 205 controls the selecting switch 206, the select- 
ing switch 207, and the selecting switch 208 in the 
above-described manner. Thus, at step S136, the mov- 
ing object connecting process portion 210 performs a 
moving object connecting process. When the deter- 
mined results represent that the current click data is a 
non-continuous click, the process determining portion 

205 controls the selecting switch 206, the selecting 
switch 207, and the selecting switch 208 in the above- 
described manner. At step S1 36, the moving object con- 

X necting process portion 21 0 performs the moving object 
connecting process. When the determined results rep- 
resent that the current click data is a non-continuous 
click, the process determining portion 205 controls the 
selecting switch 206 in the above-described manner. 
Thusi at step S1 34, the object number assigning portion 
209 performs a new object number assigning process. 
[0386] In other words, when the determined result at 
step ST 33 represents that the current click data is a non- 
continuous click, the flow advances to step S134. At 
step S1 34, the object number assigning portion 209 as- 
signs a new object number to the current click data. 
Thereafter, the flow returns to step S1 31 . 
[0387] In more reality, as shown in Fig. 42 (A), when 
an object number assigned to the immediately preced- 
ing click data CL1 denoted by a solid X mark is for ex- 
ample "0", if the current click data CL2 denoted by a dot- 
ted X mark in Fig. 42 (A) (namely, click data that has not 
assigned an object number) is determined as a non-con- 
Jtinuous click, the object number assigning portion 209 

) assigns a new object number to the current click data 
CL2 denoted by a solid X mark in Fig. 42 (B) (in this 
example the new object number is "I"), 
[0388] In contrast, when the determined result at step 
S133 represents that the current click data is a contin- 
uous click and a still click, if the immediately preceding 
click is a still click and the current click position is con- 
tained in a picture area corresponding to an object 
number assigned to the immediately preceding click da- 
ta or the current click position is close to the picture area, 
the still object connecting process portion 211 deter- 
mines that the current click is a click for the same object 
picture as the immediately preceding click. Thus, at step 
S134, the still object connecting process portion 211 
performs a still object connecting process for assigning 
the same object number as the immediately preceding 
click data for the current click data. 
[0389] In other words, as shown in a flow chart shown 
in Fig. 43, at step S1 51 , the still object connecting proc- 



ess portion 211 determines whether or not the immedi- 
ately preceding click data is a continuous click and a still 
click. When the determined result at step S161 repre- 
sents that the immediately preceding click data is a con- 
5 tinuous click and a still click, the flow advances to step 
S162. In contrast, when the determined result at step 
S161 represents the immediately preceding click is not 
a continuous click and a still click, the flow advances to 
step S164. 

10 [0390] When the determined result at step S1 61 rep- 
resents that the immediately preceding click data is not 
a continuous and still click, at step S164, the still object 
connecting process portion 211 assigns a new object 
number to the current click data in the same manner as 

*5 was described with reference to Figs. 42 (A) and (B). 
Thereafter, the flow advances to step S137 shown in 
Fig. 38. 

[0391 ] In contrast, when the determined result at step 
S161 represents that the immediately preceding click 
20 data is a continuous click and a still click, the flow ad- 
vances to step S1 62. At step S162, the still object con- 
necting process portion 211 obtains the spatial distance 
between the current click position and the picture area 
corresponding to the object number assigned to the im- 
25 mediately preceding click data. When the current click 
position is contained in the picture area corresponding 
to the object number assigned to the immediately pre- 
ceding click data or the current click position is close to 
the picture area, the still object connecting process por- 
30 tion 211 determines that the current click data is click 
data for the same object picture as the immediately pre- 
ceding click. In contrast, when the current click position 
is not contained in the object picture area corresponding 
to the object number assigned to the immediately pre- 
ss ceding click data and the current click position is farf rom 
the object picture area, the still object connecting proc- 
ess portion 21 1 determines that the current click data is 
click data for a different object picture from the immedi- 
ately preceding click. When the determined result at 
40 step S1 62 represents that the current click data is click 
data for the same object picture as the immediately pre- 
ceding click, the flow advances to step S1 63. In contrast, 
when the determined result at step S1 62 represents that 
the current click data is click data for a different object 
45 picture from the immediately preceding click, the flow 
advances to step S164. 

[0392] When the determined result at step S1 62 rep- 
resents that the current click data is click data for a dif- 
ferent object picture from the immediately preceding 

so click, the flow advances to step S1 64. At step S1 64, the 
still object connecting process portion 211 assigns a 
new object number to the current click data. Thereafter, 
the flow advances to step S1 37 shown in Fig. 38. 
[0393] I n contrast, when the determined result at step 

55 S1 62 represents that the current click data is click data 
for the same object picture as the immediately preced- 
ing click, at step S163, the still object connecting proc- 
ess portion 211 performs a still object connecting proc- 
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ess for assigning the same object number as the imme- 
diately preceding click data to the current click data. 
[0394] In more reality, as shown in Fig. 42 (C), when 
an object number assigned to the immediately preced- 
ing click data CL1 denoted by a solid X mark is "0", if 5 
the current click data CL2 denoted by a dotted X mark 
shown in Fig. 42 (C) (namely, the current click data has 
not been assigned an object number) is determined as 
a continuous click and a still click, the immediately pre- 
ceding click is a still click, and the current click position 10 
is contained in the picture area corresponding to an ob- 
ject number assigned to the immediately preceding click 
data or the current click position is close to the picture 
area, the still object connecting process portion 211 as- 
signs the same object number (in this example, "0") as 13 
the immediately preceding click data to the current click 
data CL2 denoted by the solid X mark shown in Fig. 42 

(b). 

[0395] Afterthe still object connecting process portion 
211 assigns the same object number as the immediately 20 
preceding click data to the current click data, the flow 
advances to step S137 shown in Fig. 38. 
[0396] When the determined result at step S1 33 rep- 
resents that the current click data is a continuous click 
and a moving click, the immediately preceding click is a 25 
moving click, and the feature of the picture in the vicinity 
of the current click position is contained in the feature 
of the picture area (16x16 pixels) corresponding to the 
object number assigned to the immediately preceding 
click or the former is close to the latter, the moving object 30 
connecting process portion 210 determines that the 
click is a click for the same object picture as the imme- 
diately preceding click. At step S136, the moving object 
connecting process portion 210 performs a moving ob- 
ject connecting process for assigning the same object 35 
number as the immediately preceding click data to the 
current click data. 

[0397] In other words, when the determined result at 
step S1 33 represents that the current click data is a con- 
tinuous click and a moving click, as shown in Fig. 44, at 40 
step S171 , the moving object connecting process por- 
tion 21 0 determines whether or not the immediately pre- 
ceding click data is a continuous click and a moving 
click.. When the determined result at step S171 repre- 
sents that the immediately preceding click data is a con- 45 
tinuous click and a moving click, the flow advances to 
step S172. In contrast, when the determined result at 
step S1 71 is not a continuous click and a moving click, 
the flow advances to step S1 74. 

[0398] When the determined result at step S1 71 rep- so 
resents that the immediately preceding click data is not 
a continuous click and a moving click, the flow advances 
to step S1 74. At step S1 74, the moving object connect- 
ing process portion 210 assigns a new object number 
to the current click data in the same manner described 55 
with reference to Figs. 41 (A) and (B). Thereafter, the 
flow advances to step S137 shown in Fig. 38. 
[0399] When the determined result at step S1 71 rep- 



resents that the immediately preceding click data is a 
continuous click and a moving click, the flow advances 
to step S172. At step S172, the moving object connect- 
ing process portion 21 0 obtains the feature of the picture 
area (16x16 pixels) in the vicinity of the current click 
position and the feature of the picture area correspond- 
ing to the object number assigned to the immediately 
preceding click. When the feature of the picture area in 
the vicinity of the current click position is contained in 
the feature of the picture area corresponding to the ob- 
ject number assigned to the immediately preceding click 
or the former is close to the latter, the receiving process 
portion 21 determines that the click is a click for the 
same object picture as the immediately preceding click. 
In contrast, when the feature of the picture area in the 
vicinity of the current click position is not contained in 
the feature of the picture area corresponding to the ob- 
ject number assigned to the immediately preceding click 
and the former is far from the latter, the moving object 
connecting process portion 21 0 determines thatthe cur- 
rent click data is click data for a different object picture 
from the immediately preceding click. In this case, the 
feature of the picture area is for example a color (an av- 
erage color, a typical color, orthe like), a histogram, and 
a pattern of the local area (16x16 pixels) in the vicinity 
of a click position. When the same object number is as- 
signed to a plurality of moving clicks, it means that an 
object is tracked among those click data. When the de- 
termined result at step S1 72 represents that the current 
click data is click data for the same object picture as the 
immediately preceding click, the flow advances to step 
S173. In contrast, when the determined result at step 
S1 72 represents that the current click data is clock data 
for a different object picture from the immediately pre- 
ceding click, the flow advances to step S174. 
[0400] When the determined result at step S1 72 rep- 
resents that the current click data is click data for a dif- 
ferent object picture from the immediately preceding 
click, the flow advances to step S1 74. At step S1 74, the 
moving object connecting process portion 210 assigns 
a new object number to the current click data in the 
above-described manner. Thereafter, the flow advances 
to step S 1 37 shown in Fig. 38. 

[0401] When the determined result at step S1 72 rep- 
resents that the current click data is click data for the 
same object picture as the immediately preceding click, 
the flow advances to step S1 73, At step S1 73, the mov- 
ing object connecting process portion 210 assigns the 
same object number as the immediately preceding click 
data to the current click data. 

[0402] In more reality, when an object number as- 
signed to the immediately preceding click data CL1 de- 
noted by a solid X mark shown in Fig. 42 (E) is for ex- 
ample "0", if the current click data CL2 denoted by a dot- 
ted X mark shown in Fig. 42 (E) is determined as a con- 
tinuous click and a moving click and the feature of the 
picture in the vicinity of the current click position is con- 
tained in the feature of the object picture corresponding 
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to the object number assigned to the immediately pre- 
ceding click or the former is close to the latter, the mov- 
ing object connecting process portion 210 assigns the 
same object number (in this example, "0") as the imme- 
diately preceding click data to the current click data CL2 
denoted by the solid X mark shown in Fig. 42 (F). 
[0403] After the moving object connecting process 
portion 21 0 has assigned the same object number as 
the immediately preceding click data to the current click 
data at step S1 73, the flow advances to step S137. 
[0404] When the flow advances from step S135 to 
S137 shown in Fig, 38, the object picture extracting por- 
tion 213 extracts a still object picture, a moving object 
picture, and another background picture from input pic- 
ture data corresponding to the click data of past several 
frames assigned object numbers and stored in the ob- 
ject number memory 212 and picture data of past sev- 
eral frames stored in the picture memory 20. In other 
words, it seems that a still object picture is contained in 
"j a picture portion with a high still click data density. At 
that point, the object picture extracting portion 213 ob- 
tains a still click data density corresponding to past sev- 
eral frames assigned object numbers, obtains a domi- 
nant object number of a picture portion with the high still 
click density, forms the shape of an object correspond- 
. ingtothe distribution of click data assigned the dominant 
object number, and extracts a picture in the shape of the 
object as a still object picture from the picture data. 
[0405] When the flow advances from step S136 to 
step S1 37, the object picture extracting portion 21 3 per- 
forms a pattern matching operation for pictures of the 
frames in the vicinity of click positions of moving click 
data assigned the same object number, performs a mo- 
tion compensation for the pictures corresponding to the 
matching result, obtains a dominant object number of 
the pattern matched picture area with a high click den- 
sity, forms the shape of an object corresponding to the 
distribution of click data assigned the dominant object 
number, and extracts a picture in the shape of the object 
) as a moving object picture from the picture data. . 
• [0406] At step S1 37, the object picture extracting por- 
tion 21 3 treats a picture portion with a low still click den- 
sity or low moving click data as the current background 
picture. In other words, the object picture extracting por- 
tion 21 3 treats a picture portion other than the still object 
picture and the moving object picture that have been ex- 
tracted from picture data as a background picture. 
[0407] Next, the process at step S137 will be de- 
scribed in detail with reference to a flow chart shown in 
Fig. 45. First of all, at step S181 , the object picture ex- 
tracting portion 213 captures click data of several past 
frames assigned object numbers and picture data cor- 
responding thereto . Thereafter, at step S1 82, the object 
picture extracting portion 213 categorizes the click data 
as still clicks and moving clicks. When the flow advances 
from step S1 35 to step S1 37 in Fig. 38, the flow advanc- 
es from step S1 82 to step S184 in Fig. 45. In contrast, 
whsn the flow advances from step S136 to step S1 37 in 



Fig. 38, the flow advances from step S1 82 to step S1 84 
in Fig. 45. 

[0408] When the flow advances from step S135 to 
step S137 in Fig. 38, the flow advances from step S1 82 
s to step S184 in Fig. 45. At step S184, the object picture 
extracting portion 213 obtains a still click data density 
of each still click assigned an object number for each 
block of 16 x 16 pixels. 

[0409] Thereafter, at step S1 85, the object picture ex- . 

io tracting portion 213 determines whether or not the still 
click density of still clicks denoted by X marks shown in 
Fig. 46 (A) for each block bk of 1 6 x 1 6 pixels denoted 
by a dotted box of the picture is equal to or larger than 
a predetermined value. 

15 [0410] In a picture transmitted to the receiving appa- 
ratus 2, a picture portion with a high still click density 
tends to contain a still object picture. Thus, when the still 
click density of a particular block is equal to or larger 
than the predetermined value, the flow advances to step 

20 S1 86. In contrast, when the still click density of a partic- 
ular block is smaller than the predetermined value, the 
flow advances to step S190. 

[0411] AtstepS186, when the still click density of the . 
block exceeds the predetermined value, as shown in 

25 Rg. 46 (E), the object picture extracting portion 21 3 ob- 
tains the most dominant object number from object num- 
bers assigned to click data of the block. Thereafter, the 
object picture extracting portion 213 combines blocks 
(BKO, BK2, BK4 S and BK5) corresponding to dominant 

30 object numbers as shown in Fig. 46 (B) and forms 
shapes of objects. The object picture extracting portion 
21 3 extracts pictures in the shapes of the objects as still 
object pictures from the picture data. After step S186, 
the flow advances to step S138 shown in Fig. 38. 

35 [0412] On the other hand, when the flow advances 
from step S1 36 to step S1 37 in Fig. 38, the flow advanc- 
es from step's 182 to step S1 83 in Fig. 45. At step S1 83, 
as shown in Fig. 46 (C), the object picture extracting por- 
tion 213 performs a pattern machining. operation for pic- 

40 tures of a plurality of past frames in the vicinity of click 
positions assigned the same object numbers of click da- 
ta as moving clicks denoted by X marks shown in Fig. 
46 (C) and performs a motion compensation forthe pic- 
tures corresponding to the matching result. 

45 [0413] Thereafter, at step S1 87, the object picture ex- 
tracting portion 213 obtains a moving click density in the 
pattern matched picture area. 

[0414] Thereafter, at step S1 88, the object picture ex- 
tracting portion 21 3 determines whether or not the mov- 
so ing click density of moving clicks denoted by X marks 
as shown in Fig. 46 (D) of the picture is equal to or larger 
than a predetermined value. 

[0415] A picture portion that has been motion com- 
pensated and that has a high moving click density tends 
55 to contain a moving object picture. Thus, when the mov- 
ing click density of a picture area that has been motion 
compensated is equal to or larger than the predeter- 
mined value, the flow advances to step S189. In con- 
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trast, when the moving click density of a picture area 
that has been motion compensated is smaller than the 
predetermined value, the flow advances to step S190. 
[0416] At step S1 89, the object picture extracting por- 
tion 21 3 obtain the most dominant object numbers from 
object numbers assigned to click data for a picture area 
with a moving click density that is equal to or larger than 
the predetermined value. Thereafter, as shown in Fig. 
46 (D), the object picture extracting portion 213 com- 
bines blocks (BK3 and BK6) corresponding to the dom- 
inant object numbers and forms shapes of objects. 
Thereafter, the object picture extracting portion 213 ex- 
tracts pictures in the shapes of the objects as moving 
object pictures from the picture data. After step S189, 
the flow advances to step S1 38 shown in Fig. 38. 
[0417] When the click density is smaller than the pre- 
determined value at step S1 85 and step S1 88, the flow 
advances to step S1 90. At step S1 90, the object picture 
extracting portion 213 treats a picture portion with a low 
still click density or a low moving click density as a back- 
ground picture area of the current picture. In other 
words, the object picture extracting portion 213 treats a 
picture portion otherthan a still object picture and a mov- 
ing object picture that have been extracted from picture 
data as a background picture. After step S190, the flow 
advances to step S138 shown in Fig. 38. 
[0418] Afterthe object picture extracting portion 213 
has extracted a still object picture, a moving object pic- 
ture, and a background picture from picture data, the 
flow advances to step S138 shown in Fig. 38. At step 
S138, the object picture extracting portion 213 deter- 
mines whether or not the object extracting process is 
completed. When the determined result at step S138 is 
No, the flow returns to step S131 . When the determined 
result at step S1 38 is Yes, the object picture extracting 
portion 213 completes the object extracting process. 
[041 9] In the above-described process, the object ex- 
tracting portion 1014 of the transmitting apparatus 1 
shown in Fig. 29 can extract a still object picture, a mov- 
ing object picture, and a background, picture corre- 
sponding to click data corresponding to the click opera- 
tion performed by the user of the receiving apparatus 2. 
[0420] In the embodiment shown in Fig. 28, a plain 
picture area whose activity is small (namely, that does 
not have a significance as a picture) is treated as a backr 
ground picture. For the background picture, the spatial 
resolution is not improved. Alternatively, a background 
picture may be extracted corresponding to click data, 
transmitted from the receiving apparatus 2 so as to im- 
prove the spatiaj resolution thereof. 
[0421 ] In such a case, the background picture extract- 
ing portion 1013 shown in Fig. 29 extracts a background 
picture corresponding to dick data transmitted from the 
receiving apparatus 2. The transmitting process portion 
1016 transmits the background picture so that the spa- 
tial resolution thereof is improved in the same manner 
that the spatial resolution of an object . picture is im- 
proved. 



[0422] In such a case, as shown in Fig. 47, the back- 
ground picture flag memory 74 shown in Fig. 13 is added 
to the combining process portion 1022 shown in Fig. 31 . 
The structure of the combining process portion 1022 

5 shown in Fig. 47 is the same as the structure of the com- 
bining process portion 1022 shown in Fig. 35 except for 
the background picture flag memory 74. As in the struc- 
ture shown in Fig. 13, in the structure shown in Fig. 47, 
when the background picture writing portion 71 writes a 

to background picture with a high spatial resolution to the 
background picture memory 73, the background picture 
flag stored at an address of the background picture flag 
memory 74 corresponding to each pixel that composes 
the background picture is changed from "0" to "1". In 

*5 other words, when the background picture writing por- 
- tion 71 writes background picture data to the back- 
ground picture memory 73, the background picture writ- 
ing portion 71 references the background picture flag 
memory 74. When the background picture flag is "1" 

20 (namely, when the background picture memory 73 has 
stored background picture data with a high spatial res- 
olution), the background picture writing portion 71 does 
not write background picture data with a low spatial res- 
olution to the background picture memory 73. Whenever 

25 background picture data is supplied to the background 
picture writing portion 71 ; the background picture data 
is written to the background picture writing portion 71 . 
However, when the background picture memory 73 has 
stored background picture data with a high spatial res- 

30 olution, the background picture writing portion 71 does 
not write background picture data with a low spatial res- 
olution to the background picture memory 73. Thus, 
whenever background picture data with a high spatial 
resolution is supplied to the background picture writing 

35 portion 71 , the number of background pictures with a 
high spatial resolution increases in the background pic- 
ture memory 73. 

[0423] In the example, when the combining portion 
1077 receives click data from the click data inputting 

*o portion 1024 shown in Fig. 31, the combining portion 
1077 reads object picture data and background picture 
data that contain the coordinate position of a considered 
point contained in the click data from the background 
picture memory 73 and the object memory 75 and sup- 

4* plies the obtained data to the sub window memory 79. 
[0424] Next, a technology for determining (recogniz- 
ing) a change of an interested object area of the user of 
the receiving apparatus 2 and categorizing each inter- 
ested object area will be described. 

so [0425] Various types of pictures were analyzed to de- 
termine whether or not an interested object area of the 
user has changed and categorize each interested object 
area. The analyses show the following results. 
[0426] First, an area of an interested object of a per- 

55 son (user) is an area unit that has a significance (for 
example, an object). 

[0427] Second, when an interested object of the user 
is changed, each area unit that has a significance is 
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changed. 

[0428] Third, when an interested object of the user is 
changed, the input time period necessary, for designat- 
ing the interested object of the user (for example, click 
operation) tends to becomes long. 
[0429] Fourth, wt.en an interested object of the user 
is changed, the spatial distance between input positions 
at which the user designates an interested object area 
(for example, click operation) tends to relatively become 
long. 

[0430] Thus, Fig. 48 shows an example of the struc- 
ture of the transmitting apparatus 1 that obtains the input 
time interval and the input position distance correspond- 
ing to click data that is input by the user of the receiving 
apparatus 2, determines whether or not an interested 
object area of the user of the receiving apparatus 2 has 
changed in consideration of the analyzed results (1) to 
(4), and categorizes interested object areas. The trans- 
mitting apparatus 1 shown in Fig. 48 is a modification of 

j the transmitting apparatus 1 shown in Fig. 28. For sim- 
plicity, in Fig. 48, similar portions to those in Fig. 29 are 
denoted by similar reference numerals and their de- 
scription is omitted. In other words, the structure of the 
transmitting apparatus 1 shown in Fig. 48 is basically 
the same as the structure of the transmitting apparatus 
1 shown in Fig. 29 except that a change determining 
and categorizing portion 240 is newly disposed as a 
block that composes the pre-process portion 12. 
[0431] According to the embodiment shown in Fig. 48, 
the change deten- lining and categorizing portion 240 is 
disposed intheprj-process portion 12. Alternatively, the 
change determining and categorizing portion 240 may 
be disposed in the object extracting portion 1014 or the 
background picture extracting portion 1013 of the pre- 
process portion 12. Further alternatively, the change de- 
termining and categorizing portion 240 may be disposed 
independent from the pre-process portion 12. 
[0432] In this embodiment, a position represented 
)with click data is e considered point that the user is con- 

J sidering. A consi Jored point of the usermay be not click 
data. Instead, a considered point of the usermay be rec- 
ognized by detecting the view direction of the user. 
[0433] Fig. 49 shows an example of the structure of 
the change determining and categorizing portion 240 
shown in F J g. 48. 

[0434] An input picture storing portion 231 temporarily 
stores picture data that is output by the picture inputting 
portion 11. When necessary, the input picture storing 
portion 231 supplies the picture data to a click peripheral 
area extracting portion 233 and a still area and moving 
area determining portion 234. 

[0435] A click data obtaining portion 230 temporarily 
stores click data transmitted from the receiving appara- 
tus 2 through the transmission path 3 and supplies the 
stored click data to the click peripheral area extracting 
portion 233, the stiil area and moving area determining 
portion 234, an input time interval calculating portion 
237, and an input position distance calculating portion 



238. ' 

[0436] In this embodiment, the input picture storing 
portion 231 may be disposed in common with the picture 
memory 201 shown in Fig. 37. In addition, a click data 

5 storing portion 232 may be disposed in common with 
the click data memory 202 shown in Fig. 37. 
[0437] The click peripheral area extracting portion 
233 extracts a picture area corresponding to click data 
supplied from the click data storing portion 232 from the 

10 picture data supplied from the input picture storing por- 
tion 231 (the picture area is for example a local small 
block around the click position; hereinafter, the picture 
area may be referred to as click peripheral area). Data 
of the click peripheral area extracted by the click periph- 

15 era! area extracting portion 233 is sent to a click periph- 
eral area storing portion 235: After the data is stored to 
the click peripheral area storing portion 235, the data is 
sent to an interested object area categorizing portion 
236. 

20 [0438] In addition, the still area and moving area de- 
termining portion 234 performs a stiJI area and moving 
area determination using the difference between frames 
in the same manner as the embodiment shown in Fig. 
37 corresponding to the picture data supplied from the 

25 input picture storing portion 231 and the click data sup- 
plied from the click data storing portion 232. 
[0439] The click peripheral area extracting process 
and the still area and moving area determining process 
can be accomplished by the same processes described 

30 with reference to Fig. 38. Thus, the detailed description 
of those processes is omitted. In the embodiment shown 
in Fig. 49, a result of which click data is determined as 
a still click or a moving click is output as a still area and 
moving area determination in the same manner as the 

35 above-described embodiment. In addition, for example, 
a result of which a click peripheral area is determined 
as a still area or a moving area may be output. According 
to the embodiment shown in Fig. 49, for simplicity, as a 
result of a still area and moving area determination, the 
40 case of which a still click or a moving click is output will 
be described, 

[0440] The still area and moving area determined re- 
sult of the still area and moving area determining portion 
234 is sent to the input time interval calculating portion 
45 237 and the input position distance calculating portion 
238. 

[0441] When the still area and moving area deter- 
mined result represents that click data is a still click, the 
input time interval calculating portion 237 calculates the 
so time interval between the input time of the immediately 
preceding still click and the input time of the current still 
click. In this case, the time interval is calculated regard- 
less of whether or not for example a moving click takes 
place between the input time of the immediately preced- 
es ing still click and the input time of the current still click. 
Data of the time interval calculated by the input time in- 
terval calculating portion 237 is sent to an interest 
change determining portion 239. 
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[0442] When the still area and moving area deter- 
mined result represents that click data is a still click, the 
input position distance calculating portion 238 calcu- 
lates the spatial distance between the input click posi- 
tion (coordinate position) of the immediately preceding 5 
still click and the input click position (coordinate posi- 
tion) of the current still click. In this case, the spatial dis- 
tance is calculated regardless of whether or not there is 
an input position of for example a moving click between 
the input position of the current still click and the input 
position of the immediately preceding still click. The data 
of the spatial distance calculated by the input position 
distance calculating portion 238 is sent to the interest 
change determining portion 239. 
[0443] When the still area and moving area deter- 
mined result represents that click data is a still click, the 
interest change determining portion 239 determines 
whether or not an interested object of the user has 
changed corresponding to the time interval calculated 
by the input time interval calculating portion 237 and the 
spatial distance calculated by the position distance cal- 
culating portion 238. In other words, the interest change 
determining portion 239 performs a predetermined 
weighting process for the time interval and the spatial 
distance, determines whether or not the weighted time 
interval exceeds a predetermined threshold value, 
(time), and determines whether or not the weighted spa- 
tial distance exceeds a predetermined threshold value 
(distance)! When the weighted time interval exceeds the 
predetermined threshold value and / or the weighted 
spatial distance exceeds the predetermined threshold 
value, the interest change determining portion 239 de- 
termines that the interested object of the user has 
changed. The interest change determining portion 239 
transmits the interested object change determined re- 
sult to the interested object area categorizing portion 
236. 

[0444] When the determined result of the interest 
change determining portion 239 represents that the in- 
terested object of the user has not changed, the inter- 
ested object area categorizing portion 236 determines 
that the click peripheral area of the current still click is 
contained in the same picture area as an interested ob- 
ject area corresponding to the click peripheral area of 
the immediately preceding (past) still click, categorizes 
the click peripheral area of the current still click as the 
same interested object area as the click peripheral area 
of the immediately preceding still click (for example, as- 
signs the same category number to the click peripheral 
area of the current still click), and outputs the catego- 
rized result. In other words, when an interested object 
area is categorized for each object, the same object 
number is assigned in the sama manner as the above- 
described embodiment. 

[0445] In contrast, when the determined result of the 
interest change determining portion 239 represents that 
the interested object of the user has changed, the inter- 
ested object area categorizing portion 236 determines 



that the click peripheral area of the current still click is 
not contained in the interested object area correspond- 
ing to the click peripheral area of the immediately pre- 
ceding (past) still click, outputs the stored data of the 
click peripheral picture of the current still click, and re- 
sets the data that has been stored in the dick peripheral 
area storing portion 235. Thereafter, the interested ob- 
ject area categorizing portion 236 categorizes the click 
peripheral area of the current still click as an interested 
object area that is different from the click peripheral area 
of the immediately preceding still click (for example, as- 
signs a different category number to the click peripheral 
area of the current still click). In other words, when, an 
interested object area is categorized for each object, a 
new different object number is assigned in the same 
manner as the above-described case. 
[0446] When the still area and moving area deter- 
mined result represents that click data is a moving click, 
likewise, the input time interval calculating portion 237 
calculates the time interval between the inputtime of the 
immediately preceding moving click and the input time 
of the current moving click. In this case, the time interval 
is calculated regardless of whether or not for example 
a still click takes place between the inputtime of the cur- 
rent moving click and the input time of the immediately 
preceding moving click. The data of the time interval cal- 
culated by the inputtime interval calculating portion 237 
is sent to the interest change determining portion 239. 
[0447] In contrast, when the determined result of the 
still area and moving area determined result represents 
that click data is a moving click, likewise, the input po- 
sition distance calculating portion 238 calculates the 
spatial distance between the input click position of the 
immediately preceding moving click and the input click 
position of the current movingclick. In this case/thespa- 
tial distance is calculated regardless of whether or not 
for example there is an input position of a still click be- 
tween the input position of the current moving click and 
the input position of the immediately preceding moving 
click. The data of the spatial distance calculated by the 
input position distance calculating portion 238 is sent to 
the interest change determining portion 239. 
[0448] Jn addition, the interest change determining 
portion 239 determines whether or not the interested ob- 
ject of the user has changed corresponding to the time 
interval calculated by the input time interval calculating 
portion 237 and the spatial distance calculated by the 
position distance calculating portion 238 inthecase that 
the still area and moving area determined result repre- 
sents that the click data is a moving click. In other words, 
the interest change determining portion 239 performs a 
predetermined weighting process for the time interval 
and the spatial distance, determines whether or not the 
weighted time interval exceeds a predetermined thresh- 
old value (time), and determines whether or not the 
weighted spatial distance exceeds a predetermined 
threshold value (distance). When the weighted time in- 
terval exceeds the predetermined threshold value and / 
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or the weighted spatial distance exceeds the predeter- 
mined threshold value, the interest change determining 
portion 239 determines that the interested object of the 
user has changed. The interested object change deter- 
mined result of the interest change determining portion 
239 is sent to the interested object area categorizing 
portion 236. 

[0449] When the determined result of the interest 
change determining portion 239 represents that the in- 
terested object of the user has not changed, the inter- 
ested object area categorizing portion 236 determines 
that the click peripheral area of the current moving click 
is contained in the same picture area as the interested 
object area corresponding to "he click peripheral area of 
the immediately preceding (past) moving click, catego- 
rizes the click peripheral area of the current moving click 
as the same interested object area as the click periph- 
eral area of the immediately preceding (old) moving click 
Xnamely. assign the same category number to the click 
^ peripheral area of the current moving click), and outputs 
the categorized result. In other words, when an interest- 
cd object area is categorized for each object, the same 
object number is assigned. 

[0450] In contrast, when the determined result of the 
interest change determining portion 239 represents that 
the interested object of the user has changed, the inter- 
ested object area categorizing portion 236 determines 
that the click peripheral area of the current moving click 
is not contained in the interested object area corre- 
sponding to the click peripheral area of the immediately 
preceding (past) moving click, outputs the stored data 
of the click peripheral picture of the current moving click, 
and resets the data that has been stored in the click pe- 
ripheral area storing portion 235. Thereafter, the inter- 
ested object area categorizing portion 236 categorizes 
the click peripheral area of the current moving click as 
an interested object area that is different from the click 
peripheral area of the immediately preceding moving 
(click (namely, assigns a different category numberto the 
J click peripheral area of the current moving click), in other 
words, when an interested object area is categorized for 
each object, a new different object number is assigned. 
[0451] Next, with reference to a flow chart shown in 
Fig. 50, the process of the change determining and cat- 
egorizing portion 240 shown in Fig. 49 will be described. 
[0452] At step S201 , the change determining and cat- 
egorizing portion 240 receives picture data from the pic- 
ture inputting portion 11 and click data that has been 
input by the user of the receiving apparatus 2. 
[0453] Thereafter, at step S202, the picture data that 
is supplied from the picture inputting portion 11 is stored 
to the input picture storing portion 231. The click data 
obtained by the click data obtaining portion 230 is stored 
to the click data storing portion 232. 
[0454] Thereafter, at step S203, the click peripheral 
area extracting portion 233 extracts a picture area (click 
peripheral area) corresponding to the click data from the 
picture that has been stored and read from the input pic- 



ture storing portion 231 . At step S204, the click periph- 
eral area storing portion 235 stores data of the click pe- 
ripheral area that has been extracted. 
[0455] Thereafter, at step S205, the still area and 

5 moving area determining portion 234 performs a still ar- 
ea and moving area determination using the difference 
between frames in the above-described manner. 
[0456] At step. S205, when the determined result at 
step S205 represents that click data is a still click, the 

10 flow advances to step S206. In contrast, when the de- 
termined result at step S205 represents that click data 
is a moving click, the flow advances to step S212. 
[0457] When the determined result at step S205 rep- 
resents that click data is a still click, the flow advances 

15 to step S206. At step S206, the input time interval cal- 
culating portion 237 calculates the time interval between 
the input time of the immediately preceding still click and 
the input time of the current still click. The time interval 
is calculated regardless of whether or not there is for 

20 example a moving click between the input time of the 
current still click and the input time of the immediately 
preceding still click. 

[0458] Thereafter, at step S207, the input position dis- 
tance calculating portion 238 calculates the spatial dis- 

25 tance between the input click position (coordinate posi- 
tion) of the immediately preceding still click and the input 
click position (coordinate position) of the current still 
click. In this case, the spatial distance is calculated re- 
gardless of whether or not there is for example an input 

30 position of a moving click between the input position of 
the current still click and the input position of the imme- 
diately preceding still click: 

[0459] At step S208, the interest change determining 
portion 239 determines whether or not the interested ob- 

35 ject of the user has changed corresponding to the time 
interval calculated at step S206 and the spatial distance 
calculated at step S207. In other words, as was de- 
scribed above, the interest change determining portion 
239 performs a predetermined weighting process forthe 

40 time interval and the spatial distance, determines 
whether or not the weighted time interval exceeds a pre- 
determined threshold value (time), and determines 
whether or not the weighted spatial distance exceeds a 
predetermined threshold value (distance). When the 

45 weighted time Interval exceeds the predetermined 
threshold value and / or the weighted spatial distance 
exceeds the predetermined threshold value, the interest 
change determining portion 239 determines that the in- 
terested object of the user has changed. When the de- 

50 termined result at step S208 represents that the inter- 
ested object has changed, the flow advances to step 
S209. When the determined result at step S209 repre- 
sents that the interested object has not changed, the 
flow advances to step S211 . 

55 [0460] When the determined result at step S208 rep- 
resents that the interested object of the user has not 
changed, the flow advances to step S21 1 . At step S21 1 , 
the interested object area categorizing portion 236 de- 
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termines that the click peripheral area of the current still . 
click is contained in the picture area that is the same as 
the interested object area corresponding to the click pe- 
ripheral area of the immediately preceding (old) still click 
and categorizes the click peripheral area of the current s 
still click as the sar ie interested object area as the click 
peripheral area of the immediately preceding still click 
(namely, assigns the same category number to the click 
peripheral area of the current still click). In other words, 
when an interested object area is categorized for each 
object, the same object number is assigned in the same 
manner as the above-described embodiment. After step 
S211 , the flow advances to step S21 8. 
[0461] In contrast, when the determined result at step 
S208 represents that the interested object of the user 
has changed, the flow advances to step S209. At step 

5209, the interested object area categorizing portion . 
236 determines that the click peripheral area of the cur- 
rent still click is not contained in the interested object 
area corresponding to the click peripheral area of the 
immediately preceding (old) still click, outputs the stored 
data of the click peripheral picture of the current still 
click, and resets the stored data. Thereafter, at step 

5210, the interested object area categorizing portion 
236 categorizes the click peripheral area of the current 
still click as a different interested object area from the 
click peripheral area of the immediately preceding still 
click (for example, assigns a different category number 
to the click peripheral area of the current still click). In 
other words, whe i an interested object area is catego- 
rized for each object, a new different object number is 
assigned in the same manner as the above-described 
embodiment. After step S21 1 , the flow advances to step 
S218. In contrast, when the determined result at step 
S205 represents click data is a moving click, the flow 
advances to step S212. At step S212, the interest 
change determining portion 239 calculates the time in- 
terval between the input time of the immediately preced- 
ing moving click and the input time of the current moving 
click. In this case, the time interval is calculated regard- 
less of whether or not there is for example a still click 
between the input time of the current moving click and 
the input time of the immediately preceding moving 
click. 

[0462] T.iereafter, at step S213, the interest change 
determining portion 239 calculates the spatial distance 
between the input click position of the immediately pre- 
ceding moving click and the input click position of the 
current moving click. In this case, the special distance 
is calculated regardless of whether or not there is for 
example an input position of a still click between the in- 
put position of the current moving click and the input po- 
sition of the immediately preceding moving click. 
[0463] Thereafter, at step S214, the interest change 
determining portion 239 determines whether or not the . 
interested object of the user has changed correspond- 
ing to the time interval calculated at step S212 and the 
spatial distance calculated at step S21 3. In other words, 



the interest change determining portion 239 performs a 
predetermined weighting process for the time interval 
and the special distance, determines whether or not the 
weighted time interval exceeds a predetermined thresh- 
old value (time), and determines whether or not the 
weighted spatial distance exceeds a predetermined 
threshold value (distance). When the weighted time in- 
terval exceeds the predetermined threshold value and / 
or the weighted spatial distance exceeds the predeter- 
mined threshold value, the interest change determining 
portion 239 determines that the interested object of the 
user has changed. When the determined result at step 
S21 4 represents that the interested object has changed, 
the flow advances to step S215. When the determined 
result at step S214 represents that the interested object 
has not changed, the flow advances to sep S217. 
[0464]. When the determined result at step S21 4 rep- 
resents that the interested object of the user has not 
changed, the flow advances to step S21 7. At step S21 7, 
as was described above, the interested object area cat- 
egorizing portion 236 determines that the click periph- 
eral area of the current moving click is contained in. the 
same picture area as the interested object area corre- 
sponding to the click peripheral area of the immediately 
preceding (past) moving click and categorizes the click 
peripheral area of the current moving click as an inter- 
ested object area corresponding to the click peripheral 
area of the immediately preceding moving click (for ex- 
ample, assigns the same category number to the click 
peripheral area of the current moving click), in other 
words, when an interested object area is categorized for 
each object, the same object number is assigned in the 
same manner as the above-described embodiment. Af- 
ter step S21 7, the flow advances to step S21 8. 
[0465] In contrast, when the determined result at step 
S214 represents that the interested object of the user 
has changed, the flow advances to step S215. At step 
S215, the interested object area categorizing portion 
236 determines that the click peripheral area of the cur- 
rent moving click is not contained in an interested object 
area corresponding to the click peripheral area of the 
immediately preceding (past) moving click, outputs the 
stored data of the click peripheral picture of the current 
moving click, and resets the stored data. Thereafter, at 
step S21 6, the interested object area categorizing por- 
tion 236 categorizes the click peripheral area of the cur- 
rent moving click as an interested object area that is dif- 
ferent from the click peripheral area of the immediately 
preceding moving click (for example, assigns a different 
category number to the click peripheral area of the cur- 
rent moving click). In other words, when an interested 
object area is categorized for each object, ai new differ- 
ent object number is assigned in the same manner as 
the above-described embodiment. After step S216, the 
flow advances to step S218. 

[0466] After steps S210, S211, S216, and S217, the 
flow advances to step S21 8. At step S21 8, the change 
determining and categorizing portion 240 determines 
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whether or not all the process has been completed. 
When the determined result at step S21 8 is No, the flow 
returns to step S201 . When the determined result at step 
S218 is Yes, the change determining and categorizing 
portion 240 completes the process shown in Fig. 50. 
[0467] Next, with reference to a flow chart shown in 
Fig. 51 , the interested object change determining proc- 
ess at steps S208 arid S214 shown in Fig. 50 will be 
described in detail. 

[0468] At step S221 , the interest change determining 
portion 239 obtains information of the time Interval.. 
Thereafter, at step S222, the interest change determin- 
ing portion 239 performs a predetermined weighting 
process for the time interval. At step S223, the interest 
change determining portion 239 obtains information of 
the spatial distance. Thereafter, at step S224, the inter- 
est change determining portion 239 performs a prede- 
termined weighting process for the spatial distance. The 
prder of step S221 and step S222 may be changed, 
j Likewise, the order of step S223 and S224 may be 
changed. The weighting process for the time interval 
may be performed by compressing the time unit {for ex- 
ample, a compression of ms/1 0). The weighting process 
for the spatial distance may be performed by compress- 
ing the pixel pitch in the horizontal direction and the ver- 
tical direction. 

[0469] Thereafter, the flow advances to step S225. At 
step S225, the interest change determining portion 239 
generates a three-dimensional vector using the weight- 
ed time interval (t) and the weighted spatial distance in 
the horizontal and vertical directions (x coordinate and 
y coordinate) and obtains the size of the three-dimen- 
sional vector. The size of the three-dimensional vector 
is obtained by calculating the Euclidean distance, be- 
tween the current input point and the immediately pre- 
ceding input point in the three-dimensional space of 
which the time axis (t) is added to the x coordinate axis 
and y coordinate axis of the input position of click data. 
)Afterstep S225, the flow advances to step S226. 
J [0470] At step S226, the interest change determining 
portion 239 determines whether or not the size of the 
three-dimensional vector obtained at step S225 ex- 
ceeds a predetermined threshold value. When the de- 
termined result at step S226 represents that the size of 
the three-dimensional vector does not exceed the pre- 
determined threshold value, the flow advances to step 
S227. At step S227, the interest change determining 
portion 239 determines that the interested object of the 
user of the receiving apparatus 2 has not changed. 
When the size of the three-dimensional vector exceeds 
the predetermined threshold value, the flow advances 
to step S228. At step S228, the interest change deter- 
mining portion 239 determines that the interested object 
of the user has changed. 

[0471] In the above-described manner, the change 
determining and categorizing portion 240 accomplishes 
the user's interested object change determination and 
the categorization thereof corresponding to click data 



transmitted form the receiving apparatus 2. 
[0472] In addition/when an interested object area of 
the user of the receiving apparatus 2 is categorized, the 
categorized interested object area can be optimally 

s processed. In other words, an information amount as- 
signed to each interested object area of the user may 
be varied. For example, a large information amount may 
be assigned to a particular interested object area of the 
user. Alternatively, data of an interested object area of 

10 the user may be transmitted with priority. 

[0473] . In addition, a picture area that is read (output) 
for an interested object area 236 from the click periph- 
eral area storing portion 235 can be transmitted so that 
the spatial resolution is improved as was described 

15 above. 

[0474] In this case, even if the user of the receiving 
apparatus 2 mistakenly clicks a non-Interested object 
area, it can be prevented from being incorrectly deter- 
mined. 

20 [0475] According to the embodiment, even if one ob- 
ject that has a significance as an interested object area 
is spatially or temporally separated from another object, 
they can be categorized as one object; In addition, an 
area that has a particular significance other than an ob- 

25 ject such as a substance may be extracted. 

[0476] In addition, the embodiment described with ref- 
erence to Figs, 37 to 46 and the embodiment described 
with reference to Figs. 49 to 51 may be combined. In 
this case, at steps S206 and S207, a continuous click is 

30 determined for a still click for a continuous click in the 
above-described manner. Likewise, at steps S212 and 
S21 3, a continuous click is determined for a moving click 
for a continuous click in the above-described manner. 
[0477] Next, Fig. 52 shows an example of a third 

35 structure of the picture transmitting system shown in Fig. 
1 . In Fig. 52, similar portions to those in Fig. 28 are de- 
noted by similar reference numerals and their descrip- 
tion is omitted. In other words, the structure of the picture 
transmitting system shown in Fig. 52 is basically the 

40 same as the structure of the picture transmitting system 
shown in Fig. 28 except that the exchange station 3-3 
has a charging server 4.. 

[0478] Although a service for transmitting click data 
(or control information) from the receiving apparatus 2 

45 to the transmitting apparatus 1 and providing a picture 
having an improved spatial resolution corresponding to 
click data from the transmitting apparatus 1 to the re- 
ceiving apparatus 2 (hereinafter, this service may be re- 
ferred to as click service) may be a tree server, the serv- 

50 ice may be a non-free service. When the click service is 
a non-free service, the charging server 4 performs a 
charging process for collecting a service fee for the click 
service from the user. 

[0479] In other words, Fig. 53 shows an example of 
55 the structure of the charging server 4 shown in Fig. 52. 
[0480] Predetermined information is supplied from 
the exchange station 3-3 to a communication link estab- 
lishment detecting portion 301 . The communication link 
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establishment detecting portion 301 references the in- 
formation supplied from the exchange station 3-3. As a 
result, the communication link establishment detecting 
portion 301 detects whether or not a communication link 
has been established between a terminal unit 1 as .a 
transmitting apparatus and a terminal unit 2 as a receiv- 
ing apparatus and supplies the resultant data to a ter- 
minal unit recognizing portion 302. 
[0481 ] When the terminal unit recognizing portion 302 
receives information that represents that a communica- 
tion link has been established between the terminal 
units such as the transmitting apparatus 1 and the re- 
ceiving apparatus 2 (hereinafter, this information may 
be referred to as communication link establishment in- 
formation), the terminal unit recognizing portion 302 ref- 
erences information supplied from the exchange station 
3-3. As a result, the terminal unit recognizing portion 302 
recognizes the terminal units that have been communi- 
cation linked. In addition, the terminal unit recognizing 
portion 302 recognizes IDs assigned to the terminal 
units (hereinafter, the IDs maybe referred to as terminal 
unit IDs) and supplies the recognized terminal unit IDs 
to a click detecting portion 303. 

[0482] The click detecting portion 303 monitors data 
received through the exchange station 3-3, detects click 
data transmitted from the terminal unit with a terminal 
unit ID received from the terminal unit recognizing por- 
tion 302, and supplies the detected result and the ter- 
minal unit ID to a charging process portion 304/ 
[0483] In this case, for example, the receiving appa- 
ratus 2 transmits click data along with the local terminal 
unit ID. The click detecting portion 303 compares the 
terminal unit ID added to the click data received through 
the exchange station 3-3 and the terminal unit ID sup- 
plied from the terminal unit recognizing portion 302, rec- 
ognizes the click data transmitted from the terminal unit 
that has been communication linked, and detects the 
click data. 

[0484] Hereinafter, a set of a click data detected result 
of the click detecting portion 303 and a terminal unit ID 
is referred to as click detection information. 
[0485] When the charging process portion 304 re- 
ceives the click detection information from the click de- 
tecting portion 303, the charging, process portion 304 
updates the stored content of a charging database 305. 
In addition, the charging process portion 304 performs 
the charging process corresponding to the stored con- 
tent of the charging database 305, for example, period- 
ically (for example, once per month). 
[0486] The charging database 305 stores information 
necessary for the charging process. 
[0487] Next, with reference to a flow chart shown in 
Fig. 54, the process of the charging server 4 shown in 
Fig. 53 will be described. 

[0488] The communication link establishment detect- 
ing portion 301 monitors whether or not a communica- 
tion link between terminal units has been established 
corresponding to information supplied from the ex- 



change station 3-3. When the communication link es- 
tablishment detecting portion 301 detects that the com- 
munication link between the transmitting apparatus 1 
and the receiving apparatus 2 has been established, the 

5 communication link establishment detecting portion 301 
supplies communication link establishment information 
to the terminal unit recognizing portion 302. 
[0489] When the terminal unit recognizing portion 302 
receives the communication lin k establishment informa- 

10 tion from the communication link establishment detect- 
ing portion 301, at step S301 , the terminal unit recog- 
nizing portion 302 references the infonnation supplied 
from the exchange station 3-3. As a result, the terminal 
unit recognizing portion 302 recognizes the terminal unit 

15 ids of the terminal units (for example, the transmitting 
apparatus 1 and the receiving apparatus 2) that have 
been communication linked and supplies the terminal 
unit IDs to the click detecting portion 303. 
[0490] When the click detecting portion 303 receives 

20 the terminal unit IDs from the terminal unit recognizing 
portion 302, the click detecting portion 303 starts detect- 
ing click data that contains the terminal unit IDs. 
[0491 ] Thereafter, the flow advances to step S302. At 
step S302, the charging process portion 304 determines 

25 whether or not click data has been detected from a ter- 
minal unit that has been communication linked. When 
the determined result at step S302 represents that click 
data has not been detected from a terminal unit. that has 
been communication linked (namely, when the click de- 

30 tection information has not been supplied from the click 
detecting portion 303 to the charging process portion 
304), the flow advances to step S304, skipping step 
S303. 

[0492] In contrast, when the determined result at step 

35 S3Q2 represents that click data has been detected from 
. a terminal unit that has been communication linked 
(namely, the click detection information has been sup- 
plied from the click detecting portion 303 to the charging 
process portion 304), the flow advances to step S303. 

40 At step S303, the charging process portion 304 updates 
the stored content of the charging database 305. 
[0493] In other words, the charging database 305 
stores information about clicks such as the number of 
clicks and click time (hereinafter, this information may 

45 be referred to as click information) along with commu- 
nication time at which a terminal unit originates a call 
and starts a communication. The charging database 
305 correlatively stores the click information and the ter- 
minal unit ID of the terminal unit. At step S303, the 

so charging process portion 304 updates the click Informa- 
tion corresponding to the terminal unit ID contained in 
the click detection information corresponding to the click 
detection information. 

[0494] After step S3 03, the flow advances to step 
55 S304. At stepS304,theterminal unit recognizing portion 
302 determines whether or not the communication link 
corresponding to the communication link establishment 
information supplied from the communication link estab- 
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lishment detecting portion 301 has been disconnected. 
[0495] Iri other words, the communication link estab- 
lishment detecting portion 301 monitors not only the es- 
tablishment of a communication link between terminal 
units, but the disconnection thereof. When a communi- 
cation link has been disconnected, the communication 
link establishment detecting portion 301 supplies the re- 
sultant information as communication link disconnection 
information to the terminal unit recognizing portion 302. 
At step S304, the terminal unit recognizing portion 302 
determines whether or not the communication link has 
been disconnected corresponding to the communica- 
tion link disconnection information. 
[0496] When the determined result at step S304 rep- 
resents that the communication link has not been dis- 
connected, the flow returns to step S302. At step S302, 
the charging server 4 repeats the similar process. 
[0497] In contrast, when the determined result at step 
3 04 represents that the communication link has been 
disconnected, the terminal unit recognizing portion 302 
controls the click detecting portion 303 to complete the 
monitoring of the click data of the terminal unit that has 
been communication linked. Thereafter, the charging 
server 4 completes the process. 

[0498] Thereafter, the charging process portion 304 
periodically references the charging database 305, per- 
forms the charging process, calculates the communica- 
tion fee and the click service fee, and transfers the fee 
from a bank account or the like of the user. 
[0499] As a click service fee, a unit fee per click may 
be designated. Corresponding to the number of clicks, 
the click service fee may be calculated. Alternatively, as 
a click service fee, a unit fee per hour may be designat- 
ed. Corresponding to the click time, the click service fee 
may be calculated. Alternatively corresponding to the 
number of clicks and the click time, the click service fee 
may be calculated. 

[0500] The above-described processes can be per- 
formed by hardware or software. When those processes 
are performed by software, a program that composes 
the software is installed to a computer, a general-pur- 
pose computer, or the like that is disposed in the trans- 
mitting apparatus 1 , the receiving apparatus 2, and so 
forth as dedicated hardware apparatuses. 
[0501] Thus, Fig. 55 shows an example of the struc- 
ture of a computer in which the program for executing 
the above-described processes has been installed. 
[0502] The program can be pre-recorded to a hard 
disk 405 or a ROM 403 as a record rhedium that is built 
. in the computer. 
[0503] Alternatively, the program can be temporally or 
permanently stored (recorded) on a removable record 
medium such as a floppy disk, a CD-ROM (Compact 
Disc Read Only Memory), an MO (Magneto optical) 
disc, a DVD (Digital Versatile Disc), a magnetic disc, or 
a semiconductor memory. The removable record medi- 
um 411 can be provided as a so-called package soft- 
ware. 



[0504] The program can be installed from the above- 
described removable record medium 411 to the compu- 
ter. Alternatively, the program can be wirelessly trans- 
ferred from a download site to the computer through a 

5 digital satellite broadcast satellite. Further alternatively, 
the program can be transferred to the computer through 
a network such as LAN (Local Area Network) or the In- 
ternet. In the computer, the program is received by a 
communicating portion 408 and installed to the hard disk 

10 405. 

[0505] The computer contains a CPU (Central 
Processing Unit) 402. An input / output interface 41 0 is 
connected to the CPU 402 through a bus 401 . When a 
command is input by the user with an input portion 407 
is such as a keyboard, a mouse, a microphone, and so 
forth to theCPU 402 through the input/output interface 
41 0, the program stored in a ROM (Read Only Memory) 
403 is executed. Alternatively, the CPU 402 loads the 
program stored in the hard disk 405 to a RAM (Random 
20 Access Memory) 404 and executes the program with the 
RAM 404. Further alternatively, the CPU 402 loads the 
program transferred from a satellite or a network and 
received by the communicating portion 408, and in- 
stalled to the hard disk 405 or read from the removable 
25 record medium 41 1 attached to a drive 409 and installed 
to the hard disk 405 to the RAM 404 and executes the 
program with the RAM 404. Thus, the CPU 402 per- 
forms the processes corresponding to the flow charts 
shown in Figs. 4, 6, 10, 14, 21 to 24, 27, 30, 32, 34, 36, 
30 38, 39, 41 , 43 to 45, 50, 51 , and 54. Alternatively, the 
CPU 402 performs the processes shown in the block 
diagrams of Figs. 3, 5, 7, 8, 11 to 13, 17, 25, 29, 31, 33, 
35, 37, 47 to 49, and 53. When necessary, the CPU 402 
outputs the processed results from an output portion 
35 406 composed of an LCD (Liquid Crystal Display), a 
speaker, and so forth through the input/ output interface 
41 0. Alternatively, the CPU 402 transmits the processed 
results from the communicating portion408or recorded 
to the hard disk 405. 
40 [0506] In the specification, process steps of the pro- 
gram that causes the computer to perform individual 
processes may not be always performed in the order of 
the flow charts. Alternatively, process steps may be ex- 
ecuted in parallel or discretely (for example, a parallel 
*s process or an object process may be performed). 

[0507] Trie program may be processed by one com- 
puter. Alternatively, the program may be distributed by 
a plurality of computers. Further alternatively, the pro- 
gram may be transferred to a remote computer and ex- 
so ecuted therewith. 

[0508] According to the embodiment of the present in- 
vention, the transmitting apparatus 1 performs hierar- 
chical encoding. Depending on hierarchical data that is 
transmitted, the temporal resolution and the spatial res- 
55 olution of a picture that is displayed by the receiving ap- 
paratus 2 are changed. Alternatively, the temporal res- 
olution and the spatial resolution of a picture displayed 
by the receiving apparatus 2 may be changed depend- 
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ing on discrete-cosine transform coefficients or quan- 
tized steps of a picture that are transmitted. 
[0509] Alternatively, the temporal resolution and the 
spatial resolution can be changed depending on an en- 
coding method of the transmitting apparatus 1 . in other 
words, when a picture is displayed with a regular tem- 
poral resolution, (the encoding portion 31) of the trans- 
mitting apparatus 1 chain-encodes the contour of an ob- 
ject thereof and obtains the average value of pixel val- 
ues (color) that compose an object as a representative 
value. The receiving apparatus 2 displays the area of 
the object with the color as the representative value. 
When a picture with an improved spatial resolution is 
displayed at the sacrifice of a temporal resolution, as 
was described above, the hierarchical encoding can be 
used. 

[0510] According to the embodiment, the spatial res- 
olution is improved at the sacrifice of the temporal res- 
olulion of a picture. In contrast, the temporal resolution 
may be improved at the sacrifice of the spatial resolu- 
tion. Information representing the selection of the sacri- 
ficed (improved) resolution can be contained in control 
information such as click data and transmitted from the 
receiving apparatus 2 to the transmitting apparatus 1 . 
[051 1 ] According to the embodiment of the present in- 
vention, the temporal resolution and the spatial resolu- • 
tion are. handled. However, according to the present in- 
vention, the resolution in the level direction (hereinafter, 
this resolution may be referred to as level resolution) 
may be handled. In other words, when the number of 
bits assigned to data is increased or decreased, the tem- 
poral resolution and the spatiaj resolution can be im- 
proved or deteriorated. In this case, as the temporal res- 
olution and the spatial resolution of a picture are 
changed, the tone thereof is changed. The level resolu- 
tion can be changed by changing the above-described 
quantizing steps. 

[0512] In addition, according to the embodiment, the 
spatial resolution of a partial area (priority range) of a 
picture is improved at the sacrifice of the temporal res- 
olution. Alternatively, the spatial resolution of the entire 
picture may be improved. 

[0513] Alternatively, the spatial resolution of a partic- 
ular part of a picture can be improved at the sacrifice of 
the spatial resolution of the rest of the picture rather than 
the sacrifice of the temporal resolution (namely, the tem- 
poral resolution is maintained), 

[0514] In addition, according to the embodiment, a 
picture is processed in such a manner that it is separat- 
ed into a background picture and an object. Alternative- 
ly, a picture may be processed without separating it. 
[0515] In addition, the present invention can be ap- 
plied to not only picture data, but audio data. In the case 
of audio data, for example, sampling frequency corre- 
sponds to the temporal resolution, whereas the number 
of bits assigned to audio data corresponds to the level 
resolution. 

[0516] In addition, the process of the change deter- 



mining and categorizing portion 240 shown in Fig. 49 
may be applied to the case that a feature amount of a 
sound (for example, a pitch, a desired part of a human 
voice, or each musical instrument) is extracted. 

5 [0517] In addition, the processes of the object extract- 
ing portions 14 and 1014 and the change determining 
and categorizing portion 240 can be applied to so-called 
object encoding. In other words, with the processes of 
the object extracting portions 14 and 1014 and the 

10 change determining and categorizing portion 240, since 
an object can be extracted, the extracted object can be 
encoded along with information that represents the con- 
tour or the area thereof corresponding to object encod- 
ing process that obtains information that represents the 

is motion of the object. The encoded data can be transmit- 
ted or recorded. 

[0518] The receiving apparatus 2 can perform the 
similar process to the process of the object extracting 
portion 1014 using click data so as to extract an object. 
20 In this case, when an object extracted by the receiving 
apparatus 2 is stored, an object database can be 
formed. 

[0519] According to the first transmitting apparatus^ 
the first transmitting method, the first record medium, 

25 and the first signal of the present invention, control in- 
formation transmitted from a receiving apparatus is re- 
ceived. Corresponding to the control information, the 
resolutions in at least two directions of the temporal di- 
rection, the spatial direction, and the level direction are 

30 controlled. Data of which the resolutions in at least two 
directions have been controlled corresponding to the 
control information is transmitted to the receiving appa- 
ratus. Thus, for example, the resolution in the spatial 
direction of a picture displayed by the receiving appara- 

35 tus can be more improved. 

[0520] According to the receiving apparatus, the re- 
ceiving method, the second record medium, and the 
second signal of the present invention, control informa- 
tion is transmitted to a transmitting apparatus that con- 

^o trols the resolutions in at least two directions of the tem- 
poral direction, the spatial direction, and the level direc- 
tion corresponding to the control information. In addi- 
tion, data of which the resolutions in at least two direc- 
tions have been controlled corresponding to the control 

45 information is transmitted from the transmitting appara- 
tus, received, and output. Thus, for example, the spatial 
resolution of a picture that is output can be more im- 
proved. 

[0521] According to the transmitting and receiving ap- 
50 paratus, the transmitting and receiving method, the third 
record medium, and the third signal, a transmitting ap- 
paratus receives control information transmitted from a 
receiving apparatus. Corresponding to the control infor- 
mation, the resolutions in at least two directions of the 
55 temporal direction, the spatial direction, and the level 
direction are controlled. Data of which the resolutions in 
at least two directions have been controlled correspond- 
ing to the control information is transmitted to the receiv- 
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ing apparatus. In addition, the receiving apparatus 
transmits control signal to the transmitting apparatus. 
Data of which the resolutions in at least two directions 
have been controlled corresponding to the control infor- 
mation is transmitted from the transmitting apparatus, 
received, and output. Thus, for example, the resolution 
in the spatial direction of a picture displayed by the re- 
ceiving apparatus can be more improved. 
[0522] According to the second transmitting appara- 
tus, the second transmitting method, the fourth record 
medium, and the fourth signal of the present invention, 
control signal transmitted from a receiving apparatus is 
received. Corresponding to the control information, data 
is categorized. Corresponding to the categorized result 
of the data, data is transmitted to the receiving appara- 
tus. Thus, for example, an area of a picture that the user 
is considering can be transmitted to the receiving appa- 
ratus regardless of whether the area is moving or still. 

i 

DESCRIPTION OF REFERENCE NUMERALS 
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RIZING PORTION 

INPUT TIME INTERVAL CALCULATING 
PORTION 

INPUT POSITION DISTANCE CALCU- 
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COMMUNICATION LINK ESTABLISH- 
MENT DETECTING PORTION 
CLICK DETECTING PORTION 
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COMMUNICATING PORTION 
REMOVABLE RECORD MEDIUM 
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OBJECT EXTRACTING PORTION 
COMBINING PROCESS PORTION ' 
CLICK DATA INPUTTING PORTION 
CLICK DATA TRANSMITTING PORTION 
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CHARGING SERVER 
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OBJECT EXTRACTING PORTION 
COMBINING PROCESS PORTION 
ENCODING PORTION 
DATA AMOUNT CALCULATING POR- 
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RECEIVING PORTION . 
STORING PORTION 

BACKGROUND PICTURE WRITING POR- 
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BACKGROUND PICTURE MEMORY 
OBJECT MEMORY 
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STILL OBJECT CONNECTING PROCESS. 
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OBJECT EXTRACTED RESULT MEMORY 
INPUT PICTURE STORING PORTION 
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Claims 



1. A transmitting apparatus for transmitting data to a 
receiving apparatus, comprising: . 

receiving means for receiving control informa- 
tion transmitted from the receiving apparatus; 
controlling means for controlling the resolutions 
in at least two directions of the temporal direc- 
tion, the spatial direction, and the level direction 
of the data transmitted to the receiving appara- 
tus corresponding to the control information; 
and 

transmitting means for transmitting the data of 
which the resolutions in at least two directions 
have been controlled corresponding to the con- 
trol information to the receiving apparatus. 

2. The transmitting apparatus as set forth in claim 1, 

wherein said controlling means controls the 
resolutions In the temporal direction and the spatial 
direction of the picture data transmitted to the re- 
ceiving apparatus corresponding to the control in- 
formation. 

3. The transmitting apparatus as set forth in claim 1 , 

wherein said transmitting means transmits the 
data to the receiving apparatus through a pre- 
determined transmission path at a predeter- 
mined transmission rate, and 
wherein said controlling means controls the 
resolutions of the data corresponding to the 
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control information so that the transmission rate 
of the data does not exceed the predetermined 
transmission rate. 

The transmitting apparatus as set forth in claim 1 , 5 

wherein the receiving apparatus outputs the 
data transmitted from said transmitting means, 
wherein the control information contains a con- 
sidered point of the data that is output to the 10 
receiving apparatus, and 
wherein said controlling means improves the 
resolutions of a considered area that contains 
the considered point of the data corresponding 
to the control information. *5 

The transmitting apparatus as set forth in claim 4, 

wherein said transmitting means transmits pic- 
ture data to the receiving apparatus through a 20 
predetermined transmission path at a predeter- 
mined transmission rate, 
wherein the receiving apparatus displays the 
picture data transmitted from said transmitting 
means, 25 
wherein the control information contains a tem- 
poral and special position of the picture data 
displayed by the receiving apparatus, and 
wherein said controlling means improves the 
spatial resolution of a considered area that con- 30 
tains the temporal and spatial position of the 
picture data and deteriorates the temporal res- 
olution corresponding to the control information 
so that the transmission rate of the picture data 
does not exceed the predetermined transmis- 35 
sion rate. 

The transmitting apparatus as set forth in claim 5, 
further comprising: 

40 

background picture data extracting means for 
extracting background picture data from the 
picture data transmitted to the receiving appa- 
ratus, 

wherein said controlling means improves the *s 
spatial resolution of the background picture da- 
ta when the temporal and spatial position con- 
tained in the control information represents the 
background picture data. 

50 

The transmitting apparatus as set forth in claim 6, 
further comprising: 

object picture data extracting means for ex- 
tracting object picture data from the picture da- 55 
ta corresponding to the difference between the 
picture data and the background picture data 
transmitted to the receiving apparatus , 



wherein said controlling means improves the 
spatial resolution of the object picture data 
when the temporal and spatial position con- 
tained in the control information represents the 
object picture data, 

8. The transmitting apparatus as set forth in claim 7, 
further comprising: 

combining means for combining the back- 
ground picture data and the object picture as 
combined data, 

wherein said transmitting means transmits the 
combined data to the receiving apparatus. 

9. The transmitting apparatus as set forth in claim 1 , 
further comprising: 

inputting means for inputting the data. 

10. The transmitting apparatus as set forth in claim 9, 

wherein the data is picture data, and 
wherein said inputting means is photographing 
means for photographing a picture arid output- 
ting the picture data. 

.11 . The transmitting apparatus as set forth in claim 1 , 
wherein the transmitting apparatus is a port- 
able telephone. 

12. The transmitting apparatus as set forth in claim 1 , 
further comprising: 

analyzing means for analyzing the preferences 
of the user of the receiving apparatus, 
wherein said controlling means controls the 
resolutions of the data corresponding to the an- 
alyzed result of said analyzing means. 

13. The transmitting apparatus as set forth in claim 12, 

wherein the receiving apparatus outputs the 
data transmitted from said transmitting means, 
wherein said controlling means contains a con- 
sidered point of the data that is output to the 
receiving apparatus, and 
wherein said analyzing means analyzes the 
preferences of the user corresponding to the 
considered point. 

14. The transmitting apparatus as set forth in claim 13, 

wherein said analyzing means has: 

feature amount extracting means for extracting 
a feature amount of a considered area that con- 
tains a considered point of the data; and 
area detecting means for detecting a predeter- 
mined area corresponding to the preference of 
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the user from the data corresponding to the fea- 
ture amount, and 

wherein said . controlling means controls the 
resolutions of the predetermined area of the da- 
ta. 

15. The transmitting apparatus as set forth in claim 14, 
further comprising: 

histogram storing means for storing a histo- 
gram of the future amount, 
wherein said area detecting means detects the 
predetermined area corresponding to the his- 
togram. 

,16. The transmitting apparatus as set forth in claim 14, 

wherein said transmitting means transmits pic- 
ture data to the receiving apparatus through a 
predetermined transmission path at a predeter- 
mined transmission rate, 
wherein the receiving apparatus displays the 
picture data transmitted from said transmitting 
means, and 

wherein said controlling means improves the 
spatial resolution of the predetermined area of 
the picture data and deteriorates the temporal 
resolution so that the transmission rate of the 
picture data does not exceed the predeter- 
mined transmission rate. 

17. The transmitting apparatus as set forth in claim 15, 
wherein said area detecting means detects an 
area having the same as or similar to the feature 
amount with the largest frequency of the histogram 
as the predetermined area. 
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18. The transmitting apparatus as set forth in claim 17, 

wherein said transmitting means transmits pic- 
ture data to the receiving apparatus through a 
predetermined transmission path at a predeter- 
mined transmission rate, 
wherein the receiving apparatus displays the 
picture data transmitted from said transmitting 
means, and 

wherein said controlling means improves the 
spatial resolution of the predetermined area of 
the picture data and deteriorates the temporal 
resolution so that the transmission rate of the 
picture data does not exceed the predeter- 
mined transmission rate. 

19. The transmitting apparatus as set forth in claim 16, 

wherein said feature amount extracting 
means extracts at least one of motion information, 
depth information, position information, color infor- 
mation, and shape information of a considered area 



that contains the considered point of the picture da- 
ta as the feature amount. 

20. The transmitting apparatus as set forth in claim 19, 
wherein said feature amount extracting 

means extracts a plurality of motion information, 
depth information, position information, color infor- 
mation, and shape information of a considered area 
that contains the considered point of the picture da- 
ta as a plurality of feature amounts, and generates 
a feature amount vector composed of the plurality 
of feature amounts. 

21. The transmitting apparatus as set forth in claim 13, 
wherein said analyzing means has: 

categorizing means for categorizing the data 
corresponding to a considered area that con- 
tains the considered point of the data, 
wherein said analyzing means analyzes the 
preference of the user corresponding to the an- 
alyzed result of said categorizing means. 

22. The transmitting apparatus as set forth in claim 21 , 

wherein the data is picture data, 
wherein said analyzing means has: 
still area and moving area determining means 
for determining whether the considered area of 
30 the picture data is a still area or a moving area; 

and 

continuity determining means for determining 
whether the considered point is continuous in 
the temporal and spatial directions of the con- 

35 sidered point, and 

wherein said categorizing means categorizes 
the picture data corresponding* to the deter- 
mined results of the still area and moving area 
determining means and said continuity deter- 

40 mining means. 

23. The transmitting apparatus as set forth in claim 22, 
further comprising: 

45 considered point storing means for storing a 

considered point that is contained in the con- 
sidered area that is still and that is continuous 
in the temporal and spatial directions and acbn- 
sidered point that is contained in the consid- 

so ered area that is moving and that is continuous 

in the temporal and spatial directions; and 
category identifier adding means for obtaining 
a category identifier added to a considered 
point stored to the said considered point storing 

55 means and adding the category identifier to the 

considered point. 

24. The transmitting apparatus as set forth in claim 23, 
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wherein in the case that the current consid- 
ered point is in the considered area that is still and 
that is continuous in the temporal and spatial direc- 
tions, when an immediately preceding considered 
point stored in said considered point storing means 
is contained in the considered area that is still and 
that is continuous in the temporal and spatial direc- 
tions, said category identifier adding means obtains 
a category identifier added to the current consid- 
ered point corresponding to the relation of the spa- 
tial positions between the current considered point 
and the area that contains the immediately preced- 
ing considered point. 

25. The transmitting apparatus as set forth in claim 24, 

wherein in the case that the current consid- 
ered point is in the considered area that is moving 
and that is continuous in the temporal and spatial 
directions : when an immediately preceding consid- 
ered point stored in said considered point storing 
means is contained in the considered area that is 
moving and that is continuous in the temporal and 
spatial directions, said category identifier adding 
means obtains a category identifier added to the 
current considered point corresponding to the sim- 
ilarity of a predetermined feature amount of the con- 
sidered area that contains the current considered 
point and that of the considered area that contains 
the immediately preceding considered point. 

26. The transmitting apparatus as set forth in claim 23, 

wherein said categorizing means categorizes 
a predetermined area of the picture data as an ob- 
ject corresponding to the preference of the user cor- 
responding to the density of considered points 
stored in said considered point storing means. 

27. The transmitting apparatus as set forth in claim 26, 

wherein said categorizing means categorizes 
a predetermined area of the picture data as an ob- 
ject corresponding to the preference of the user cor- 
responding to the density of considered points as- 
signed the same category identifier, stored in said 
considered point storing means, and contained in 
the considered area that is still. 

28. The transmitting apparatus as set forth in claim 26, 

wherein said categorizing means categorizes 
a predetermined area of the picture data as an ob- 
ject corresponding to the preference of the user cor- 
responding to the density of considered points 
stored in said considered point storing means, con- 
tained in the considered area that is moving, as- 
signed the same category identifier, and motion 
compensated. 

29. The transmitting apparatus as set forth in claim 22, 

wherein said still area and moving area deter- 



mining means determines whether a considered ar- 
ea that contains the current considered point is still 
or moving corresponding to the difference between 
the considered area that contains the considered 
point of the current frame and the considered area 
that contains the considered point of a past frame. 

30. The transmitting apparatus as set forth in claim 22, 

wherein said continuity determining means 
determines whether or not the current considered 
point is continuous in the temporal and spatial di- 
rections corresponding to the time difference be- 
tween the current considered point and a past con- 
sidered point. 

31. The transmitting apparatus as set forth in claim 26, 

wherein said controlling means improves the 
resolutions of the area categorized as the object. 

32. The transmitting apparatus as set forth in claim 22, 

wherein said continuity determining means 
determines whether or not the current considered 
point is continuous corresponding to the distances 
in the temporal direction and the spatial direction 
between the current considered point and a past 
considered point at which the same still area and 
moving area determined result as the considered 
area that contains the current considered point is . 
obtained. 

33. The transmitting apparatus as set forth in claim 32, 

wherein said categorizing means categorizes 
the picture data corresponding to weighted values 
for the distances in the temporal direction and the 
spatial direction. ' 

34. The transmitting apparatus as set forth in claim 33, 
further comprising: 

picture data storing means for storing picture 
data in the considered area that contains a consid- 
ered point that is continuous in the temporal direc- 
tion and the spatial direction. 

35. The transmitting apparatus as set forth in claim 34, 

wherein when the current considered point is 
not continuous In the temporal direction and the 
spatial direction, after the content of said picture da- 
ta storing means is read, the content is erased and 
the picture data in the considered area that contains 
the current considered point is stored to said picture 
data storing means. 

36. The transmitting apparatus as set forth in claim 35, 

wherein said controlling means improves the 
resolutions of the picture data that is read from said 
picture data storing means. 

37. The transmitting apparatus as set forth in claim 1, 
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wherein the control information is used for a 
charging process. 

38. The transmitting apparatus as set forth in claim 2, 

wherein the picture data is object encoded. 

39. A receiving apparatus for receiving data transmitted 
from -.a transmitting apparatus, comprising: 

transmitting means for transmitting control in- 
formation to the transmitting apparatus that 
controls resolutions in at least two directions of 
the temporal direction, the spatial direction, and 
the level direction of the data corresponding to 
the control information; 

receiving means for receiving the data transmit- 
ted from the transmitting apparatus, the reso- 
lutions of the data having been controlled in at 
least two directions of the data; and . 
j outputting means for outputting the data re- 

ceived by said receiving means. 

40. The receiving apparatus as set forth in claim 39, 

wherein the data is picture data, and 
wherein said outputting means is displaying 
means for displaying the picture data. 

41 . The receiving apparatus as set forth in claim 40, fur- 
ther comprisir g: 

considered point detecting means for detecting 
a considered point of the user from the picture 
data displayed by said displaying means, 
wherein said transmitting means transmits the 
considered point as the control information to 
the transmitting apparatus. 

}42. The receiving apparatus as set forth in claim 41 , 
J wherein said considered point detecting 

means detects the position designated by said des- 
ignating means as the considered point. 

43. The receiving apparatus as set forth in claim 40, fur- 
ther comprising: 

picture data storing means for storing picture 
data received by said receiving means; and 
controlling means for causing picture data 
stored in said picture data storing means to be 
displayed by said displaying means when the 
resolutions of the picture data stored in said pic- 
ture data storing means are.higherthan the res- 
olutions of the picture data received by said re- 
ceiving means. 

44. The receiving apparatus as set forth in claim 43, 

wherein said controlling means causes the 



picture data received by said receiving means to be 
overwritten to said picture data storing means the 
picture data received by said receiving means to be 
displayed by said displaying means when the res- 

5 olutions of the picture data stored in said picture da- 
ta storing means are lower than the resolutions of . 
the picture data received by said receiving means, 
the picture data stored in said picture data storing 
means corresponding to the picture data received 

10 by said receiving means. 

45. The receiving apparatus as set forth in claim 39, 

wherein the control information is used for a 
charging process. 

15 

46. A transmitting and receiving apparatus having a 
transmitting apparatus for transmitting data and a 
receiving apparatus for receiving the data, 

wherein the transmitting apparatus compris- 

20 es: 

control information receiving means for receiv- 
ing control information transmitted from the re- 
ceiving apparatus; 

25 controlling means for controlling the resolutions 

in at least two directions of the temporal direc- 
tion, the spatial direction , and the level direction 
of the data transmitted to the receiving appara- 
tus corresponding to the control information; 

30 and 

data transmitting means for transmitting the da- 
ta of which the resolutions in at least two direc- 
tions have been controlled corresponding to the- 
control information to the receiving apparatus, 

35 and 

wherein the receiving apparatus comprises: 

control information transmitting means for 
transmitting the control information; 
40 data receiving means for receiving the data 

transmitted from the transmitting appara- 
tus, the resolutions of the data having been 
controlled in at least two directions of the 
data; and 

45 outputting means for outputting the data re- 

ceived by said data receiving means. 

47. A transmitting apparatus for transmitting data to a 
receiving apparatus, comprising: 

50 

receiving means for receiving control informa- 
tion transmitted from the receiving apparatus; 
categorizing means for categorizing the data 
corresponding to the control information; and 
55 transmitting means for transmitting the data to 

the receiving apparatus corresponding to the 
categorized result of the data. 
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48. The transmitting apparatus as set forth in claim 47, 

wherein the data is picture data, 

wherein the receiving apparatus displays the 

picture data transmitted from said transmitting 

means, 

wherein the control information contains a con- 
sidered, point of picture data displayed by the 
receiving apparatus, and 
wherein said categorizing means categorizes . 
the picture data corresponding to a considered 
area that contains the considered point of the 
picture data. 

49. The, transmitting apparatus as set forth in claim 48, 
further comprising: 

still area and moving area determining means 
for determining whether or not the considered 
area of the picture data is still or moving; and 
continuity determining means for determining 
whether or not the considered point is continu- 
ous in the temporal direction and the spatial di- 
rection, 

wherein said categorizing means categorizes 
the picture data corresponding to the deter- 
mined results of said still area and moving area 
determining means and said continuity deter- 
mining means. 

50. The transmitting apparatus as set forth in claim 49, 
further comprising: 

considered point storing means for storing a 
considered point that is contained in the con- 
sidered area that is still and that is continuous 
in the temporal direction and the spatial direc- 
tion and a considered point that is contained in 
the considered area that is moving and that is 
continuous in the temporal direction and the 
spatial direction; and 

category identifier adding means for obtaining 
a category identifier added to a considered 
point stored in said considered point storing 
means and adding the category identifier to the 
considered point. 

51. The transmitting apparatus as set forth in claim 50, 

wherein in the case that the current consid- 
ered point is in the considered area that is still and 
that is continuous in the temporal and spatial direc- 
tions, when an immediately preceding considered 
point stored in said considered point storing means 
is contained in the considered area that is still and 
that is continuous in the temporal and spatial direc- 
tions, said category identifier adding means obtains 
a category identifier added to the current consid- 
ered point corresponding to the relation of the spa- 



tial positions between the current considered point 
and the area that contains the immediately preced- 
ing considered point. 

5 52. The transmitting apparatus as set forth in claim 50, 
wherein in the case that the current consid- 
ered point is in the considered area that is moving 
and that is continuous in the temporal and spatial 
directions, when an immediately preceding consid- 

10 ered point stored in said considered point storing 
means is contained in the considered area that is 
moving and that is continuous in the temporal and 
spatial directions, said category identifier adding 
means obtains a category identifier added to the 

*5 current considered point corresponding to the sim- 
ilarity of a predetermined feature amount of the con- 
sidered area that contains the current considered 
point and that of the considered area that contains 
the immediately preceding considered point. 

20 

53. The transmitting apparatus as set forth in claim 50, 

wherein said categorizing means categorizes 
a predetermined area of the picture data as one ob- 
ject corresponding to the density of considered 
25 points stored in said considered point storing 
means. 

54. The transmitting apparatus as set forth in claim 53, 

wherein said categorizing means categorizes 
30 a predetermined area of the picture data as one ob- 
ject corresponding to the density of considered 
points assigned the same category identifier, stored 
in said considered point storing means, and con- 
tained in the considered area that is still. 

35 

55. The transmitting apparatus as set forth in claim 53, 
wherein said categorizing means categorizes 

a predetermined area of the picture data as one ob- 
ject corresponding to the density of considered 
points stored in said considered point storing 
means, contained in the considered area that is 
moving, assigned the same category identifier and 
motion compensated. 

56. The transmitting apparatus as set forth In claim 49, 
wherein said still area and moving area deter- 
mining means determines whether a considered ar- 
ea that contains the current considered point is still 
or moving corresponding to the difference between 
the considered area that contains the considered 
point of the current frame and the considered area 
that contains the considered point of a past frame. 

57. The transmitting apparatus as set forth in claim 49, 
wherein said continuity determining means 

determines whether or not the current considered 
point is continuous in the temporal and spatial di- 
rections corresponding to the time difference be- 
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tween the current considered point and a past con- 
sidered point. 

58. The transmitting apparatus as set forth in claim 53, 

wherein said controlling means improves the 5 
resolutions of the area categorized as the object. 



59. The transmitting apparatus as set forth in claim 49, 
wherein said continuity determining means 
determines whether or not the current considered 
point is continuous corresponding to the distances 
in the temporal direction and the spatial direction 
between the current considered point and a past 
considered point at which the same still area and 
moving area determined result as the considered 
area that contains the current considered point is 
obtained. 

£0. The transmitting apparatus as set forth in claim 59, 
wherein said categorizing means categorizes 
the picture data corresponding to weighted values 
for the distances in the temporal direction and the 
spatial direction. 

61. The transmitting apparatus as set forth in claim 59, 
further comprising: 

picture data storing means for storing picture 
data in the considered area that contains a consid- 
ered point that is continuous in the temporal direc- 
tion and the spatial direction. 

62. The transmitting apparatus as set forth in claim 61 ,- 

wherein when the current considered point is 
not continuous in the temporal direction and the 
spatial direction, after the content of said picture da- 
ta storing means is read, the content is erased and 
the picture data in the considered area that contains 
the current considered point is stored to said picture 
) data storing means. 

63. The transmitting apparatus as set forth in claim 62, 

wherein said controlling means improves the 
resolutions of the picture data that is read from said 
picture data storing means. 

64. The transmitting apparatus as set forth in claim 47, 

wherein the control information is used for a 
charging process. 

65. The transmitting apparatus as set forth in claim 48, 

wherein the picture data is object encoded. 

66. A transmitting method for transmitting data to a re- 
ceiving apparatus, comprising the steps of: 

receiving control information transmitted from 

the receiving apparatus; 

controlling the resolutions in at least two direc- 



tions of the temporal direction, the spatial direc- 
tion, and the level direction of the data trans- 
mitted to the receiving apparatus correspond- 
ing to the control information; and . 
transmitting the data of which the resolutions in 
at least two directions have been controlled cor- 
responding to the control information to the re- 
ceiving apparatus. 

10 67. A receiving method for receiving data transmitted 
from a transmitting apparatus, comprising the steps 
of: 

transmitting control information to the transmit- 
1 5 ting apparatus that controls resolutions in at 

least two directions of the temporal direction, 
the spatial direction, and the level direction of 
the data corresponding to the control informa- 
tion; 

20 receiving the data transmitted from the trans- 

mitting apparatus, the resolutions of the data 
having been controlled in at leasttwo directions 
of the data; and 

outputting the data received at the receiving 
25 step. 

68. A transmitting and receiving method having a proc- 
ess of a transmitting apparatus for transmitting data 
and a process of a receiving apparatus for receiving 
30 the data, 

wherein the process of the transmitting appa- 
ratus comprises the steps of: 

receiving control information transmitted from 
35 the receiving apparatus; 

controlling the resolutions in at least two direc- 
tions of the temporal direction, the spatial direc- 
tion, and the level direction of the data trans- 
mitted to the receiving apparatus correspond- 
ing to the control information; and 
transmitting the data of which the resolutions in 
atleasttwo directions havebeen controlled cor- 
responding to the control information to the re- 
ceiving apparatus, and 

wherein the process of the receiving apparatus 
comprises the steps of: 

transmitting the control information; 
receiving the data transmitted from the 
transmitting apparatus, the resolutions of 
the data having been controlled in at least 
two directions of the data; and 
outputting the data received at the data re- 
ceiving step. 

A transmitting method for transmitting data to a re- 
ceiving apparatus, comprising the steps of: 
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receiving control information transmitted from 
the receiving apparatus; 
categorizing the data corresponding to the con- 
trol information; and 

transmitting the data to the receiving apparatus 
correspon ding to the categorized result of the 
data. 

70. A record medium for recording a program that caus- 
es a computer to perform a transmitting process for 
transmitting data to a receiving apparatus, the 
transmitting process comprising the steps of: 

receiving control information transmitted from 
the receiving apparatus; . 
controlling the resolutions in at least two direc- 
tions of the temporal direction, the spatial direc- 
tion, and the level direction of the data trans- 
mitted to the receiving apparatus correspond- 
ing to the control information; and 
transmitting the data of which the resolutions in 
at least two directions have been controlled cor- 
responding to the control information to the re- 
ceiving apparatus. 

71 . A record medium for recording a program that caus- 
es a computer to perform a receiving process for 
receiving data transmitted from a transmitting ap- 
paratus, the receiving process comprising the steps 
of: 

transmitting control information to the transmit- 
ting apparatus that controls resolutions in at 
least two directions of the temporal direction, 
the spatial direction, and the level direction of 
the data corresponding to the control informa- 
tion; 

receiving the data transmitted from the trans- 
mitting apparatus, the resolutions of the data 
having boen controlled in at least two directions 
of the data; and 

outputting the data received at the receiving 
step. 

72. A reco.d medium for recording a program that caus- 
es a computer to perform a transmitting process of 
a transmitting apparatus for transmitting data and a 
receiving process of a receiving apparatus for re- 
ceiving the data, 

wherein the transmitting process of the trans- 
mitting apparatus comprises the steps of: . 

receiving control information transmitted from 
the receiving apparatus; 

controlling the resolutions in at least two direc- 
tions of the temporal direction, the spatial direc- 
tion, and the level direction of the data trans- 
mitted to the receiving apparatus correspond- 



10 



15 



ing to the control information; and 
transmitting the data of which the resolutions in 
at least two directions have been controlled cor- 
responding to the control information to the re- 
ceiving apparatus, and 

wherein the receiving process of the receiving 
apparatus comprises the steps of: 

transmitting the control information; 
receiving the data transmitted from the 
transmitting apparatus, the resolutions of 
the data having been controlled in at least 
two directions of the data; and 
outputting the data received at the data re-, 
ceiving step. 



73. A record medium for recording a program that caus- 
es a computer to perform a transmitting process for 
transmitting data to a receiving apparatus, the 
20 transmitting process comprising the steps of: 



receiving control information transmitted from 
the receiving apparatus; 

categorizing the data corresponding to the con^ 
trol information; and 

transmitting the data to the receiving apparatus 
corresponding to the categorized result of the 
data. 
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74. A signal for containing a program that causes a 
computer to perform a transmitting process for 
transmitting data to a receiving apparatus, the 
transmitting process comprising the steps of: 

receiving control information transmitted from 
the. receiving apparatus; 
controlling the resolutions in at least two direc- 
tions of the temporal direction, the spatial direc- 
tion, and the level direction of the data trans- 
mitted to the receiving apparatus correspond- 
ing to the control information; and 
transmitting the data of which the resolutions in 
at least two directions have been controlled cor- 
responding to the control information to the re- 
ceiving apparatus. 

75. A signal for containing a program that causes a 
computer to perform a receiving process for receiv- 
ing data transmitted from a transmitting apparatus, 
the receiving process comprising the steps of: 

transmitting control information to the transmit- 
ting apparatus that controls resolutions in at 
least two directions of the temporal direction, 
the spatial direction, and the level direction of 
the data corresponding to the control informa- 
tion; 

receiving the data transmitted from the trans- 
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mitting apparatus, the resolutions of the data 
having been controlled in at least two directions 
of the data; and 

outputting the data received at the receiving 
step. 5 

76. A signal for containing a program that causes a 
computer to perform a transmitting process of a 
transmitting apparatus for transmitting data and a 
receiving process of a receiving apparatus for re- 10 
ceiving the data, 

wherein the transmitting process of the trans- 
mitting apparatus comprises the steps of: 

receiving control information transmitted from *5 
the receiving apparatus; 
controlling the resolutions in at least two direc- 
tions of the temporal direction, the spatial direc- 
tion, and the level direction of the data trans- 
mitted to the receiving apparatus correspond- 20 
ing to the control information; and 
transmitting the data of which the resolutions in 
at least two directions have been controlled cor- 
responding to the control information to the re- 
ceiving apparatus, and 25 
wherein the receiving process of the receiving 
apparatus comprises the steps of: 

transmitting the control information; 
receiving the data transmitted from the 30 
transmitting apparatus, the resolutions of 
the data having been controlled in at least 
two directions of the data; and 
outputting the data received at the data re- 
ceiving step. 35 

77. A signal for containing a program that causes a 
computer to perform a transmitting process for 
transmitting data to a receiving apparatus, the 
transmitting process comprising the steps of: 40 

receiving control information transmitted from 
the receiving apparatus; 
categorizing the data corresponding to the con- 
trol information; and 45 
transmitting the data to the receiving apparatus 
corresponding to the categorized result of the 
data. 
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